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Abstract 

Information  technology  investment  has  become  a  significant  part  of  the 
Department  of  Defense’s  budget.  The  Information  Technology  Management 
Reform  Act  requires  that  government  quantitatively  evaluate  future  information 
technology  investments.  Quantitative  evaluation  of  IT  investments  present 
special  problems  for  agencies  that  do  not  generate  incoming  cash  flows. 

This  thesis  is  designed  to  examine  models  that  are  currently  being  used  in 
the  public  and  private  sector  of  the  economy  to  evaluate  Information  Technology 
investments  to  learn  which  ones  might  serve  the  needs  of  the  United  States  Air 
Force.  The  methodology  is  an  exploratory  study  based  on  Criterion-based 
Congruence  Analysis.  This  technique  is  designed  to  collect  information  and  then 
filter  it  to  surface  the  information  that  is  pertinent  to  the  research  question. 

This  research  uncovered  18  models  that  use  a  variety  of  methods  to  value 
information  technology.  Of  these  models,  five  could  currently  be  used  by  the  Air 
Force  based  on  the  availability  of  the  required  data.  These  models  are:  Cost 
Benefit  Analysis,  Cost  of  Information  Management,  IT  Spending,  Residual  Value, 
and  the  Balanced  Scorecard.  There  are  two  additional  models  that  may 
contribute  to  investment  decisions  in  very  specific  circumstances.  These  models 
are:  Business  Profitability,  and  Information  Productivity. 


AN  EXAMINATION  OF  INFORMATION  TECHNOLOGY  VALUATION  MODELS 


FOR  THE  AIR  FORCE 


I.  Introduction 


Background 

Since  the  late  1960s,  private  industry  and  government  has  invested  and 
continues  to  invest  a  significant  quantity  of  money  in  information  technology  (IT). 
Estimates  for  worldwide  annual  spending  on  computers  and  associated  products 
are  over  $1  trillion  annually  with  the  United  States’  share  of  this  spending  over 
$500  billion  (Strassmann,  1997:26).  There  are  high  expectations  for  increases  in 
productivity  and  profitability  associated  with  these  investments.  Taxpayers  and 
stockholders  both  have  the  right  to  expect  the  substantial  investments  in  IT  to 
have  positive  effects  on  the  operations  of  the  entity.  In  the  manufacturing  sector, 
productivity  increases  have  been  significant,  however,  these  increases  have  not 
been  seen  in  the  service  sector  (Brynjolfsson,  1993:68).  Government,  which  is 
primarily  a  service  based  organization,  has  also  not  shown  the  expected 
productivity  increases  from  substantial  investments  in  IT.  “Despite  spending 
more  than  $200  Billion  on  information  management  and  systems,  the 
government  has  too  little  evidence  of  meaningful  returns”  (United  States 
Government  Accounting  Office,  1994:5).  One  example  of  this  is  the  Department 
of  Defense’s  (DoD)  experience  with  Ada.  “As  a  result  of  the  investment  in  Ada, 
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the  DoD  now  possesses  code  that  is  expensive  to  maintain”  (Strassmann, 
1997:148). 

In  1996,  the  Information  Technology  Management  Reform  Act  (ITMRA) 
tasked  the  Federal  Government  and  all  subordinate  agencies,  including  the 
Department  of  Defense  (DoD),  to  quantitatively  justify  their  investments  in  IT 
(U.S.  Congress,  1996:  Section  5122  b  5). 

The  ITMRA  is  the  driving  force  behind  this  project.  This  law  directs  the 
government  to  better  evaluate  its  IT  investments  and  is  a  broad  directive  on  how 
information  and  IT  investment  should  be  handled  by  the  government. 

Specifically,  in  Section  5122,  paragraph  b3,  the  ITMRA  requires  that  minimum 
criteria  related  to  quantifiably  expressed  project  net  value,  risk  adjusted  ROI,  and 
prioritzability  of  projects  be  applied  to  any  new  IT  investments.  In  Section  5122 
(b)  5  of  the  ITMRA,  which  provides  one  of  the  bases  for  this  research,  agencies 
are  also  tasked  with  identifying  other  quantifiable  techniques  for  evaluation. 

The  Federal  Government  has  two  unique  problems  it  must  solve  to  meet 
the  intent  of  the  ITMRA.  Government  entities  operate  in  a  environment  vastly 
different  from  private  enterprises.  Most  government  organizations  are  funded 
through  budgets  and  do  not  face  the  constraint  of  generating  cash  flows  from 
operations  to  justify  new  IT  investment. 

As  the  literature  review  will  show,  many  of  the  currently  available  models 
that  measure  IT  valuation  are  based  on  revenue  generation.  This  causes 
significant  difficulty  in  applying  these  models  to  organizations  such  as  the  DoD 
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where  incoming  cash  flows  from  operations  are  non-existent.  Other  variables 
embedded  in  the  model  may  also  require  information  that  is  not  available.  This 
leads  to  the  problem  of  whether  or  not  the  data  is  available  for  each  model. 

IT  investment  is  very  difficult  to  evaluate  when  there  are  revenue 
exchanges.  This  problem  is  exacerbated  when  there  are  no  revenue  flows  to 
base  the  valuation  on,  such  as  in  a  government  entity.  The  problems  are  also 
compounded  by  using  contemporary  methods  of  asset  valuation  and  productivity 
measures  because  of  the  unique  capabilities  and  opportunities  IT  provides. 

The  second  problem  is  that  there  are  no  productivity  measures  that 
adequately  quantify  IT  productivity  at  the  firm  level.  Noted  IT  researcher  and 
practitioner  Strassmann  noted  the  problems  associated  with  measuring  IT 
productivity  by  describing  the  results  of  a  study  by  the  National  Research  Council 
in  relation  to  research  by  Brynjolfsson  and  Hitt  (Strassmann,  1997:95-96). 
Fortune  (Magnet,  1994:79),  and  Business  Week  (Gleckman  et  al,  1993:56), 
proclaimed  that  studies  by  Brynjolfsson  and  Hitt  had  finally  proven  the 
productivity  of  computers.  However,  at  the  same  time,  the  National  Research 
Council  published  a  study  that  concluded  that  “during  the  1980’s,  the  service 
sector  showed  limited  gains  in  productivity  despite  and  extraordinary  burst  of 
spending  on  IT”  (National  Research  Council,  1994:5). 

To  summarize  one  of  their  research  efforts,  Hitt  and  Brynjolfsson 
concluded  that  “while  IT  has  been  productive  for  the  average  firm,  many  firms 
undoubtedly  made  unproductive  investments  in  IT”  (Hitt  and  Brynjolfsson, 
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1996:136).  This  statement  suggests  that  organizations  haven’t  yet  been  able  to 
differentiate  which  IT  investments  lead  to  increased  productivity  and  what 
specific  factors  drive  increased  productivity. 

There  has  been  a  significant  amount  of  research  in  the  areas  of  IT 
valuation  and  productivity,  but  the  majority  of  research  is  still  based  mainly  on 
theory  and  has  not  been  extensively  tested  at  the  organizational  level.  As 
budgets  continue  to  be  cut,  the  government  must  find  better  ways  to  evaluate  its 
investments  in  IT. 

Researchers  and  practitioners  have  suggested  that  organizations  are  not 
receiving  as  many  benefits  as  they  should  from  the  investment  in  IT.  This  is 
what  led  to  the  ITMRA  requiring  quantitative  evaluation  of  the  risks  and  benefits 
of  IT  investment. 

Office  productivity  has  been  stagnant  in  the  last  several  years  while 
investment  in  IT  has  been  increasing  substantially  (Panko,  1991:199).  Panko 
observed  that  this  has  profound  implications  for  researchers  and  practitioners. 
Part  of  the  problem  with  this  data  is  how  we  measure  productivity.  The  Bureau 
of  Labor  Statistics  (BLS)  does  not  currently  collect  data  for  office  productivity 
because  of  the  difficulties  with  measurement.  The  majority  of  data  that  BLS 
analyzes  is  based  on  single  factors  of  production  such  as  a  units  of  output  per 
units  of  labor,  or  units  of  output  per  unit  of  capital.  Office  productivity  does  not 
lend  itself  well  to  such  measures. 
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Multi-factor  models,  such  as  the  capital-labor  multifactor  productivity 

model  and  the  KLEMS  (capital,  labor,  energy,  materials,  and  outside  services) 

multifactor  productivity  model,  currently  used  in  the  manufacturing  sector,  would 

better  measure  office  productivity  (Panko,  1991:196).  However,  even  this  may 

not  solve  the  measurement  problem.  The  problem  is  not  how  much  we  invest  in 

IT,  but  how  we  use  IT  (Panko,  1991:199). 

IT  investment  should  not  be  blamed  for  the  stagnant  growth  in  productivity 

for  two  reasons.  The  first  reasons  is  that 

productivity  is  difficult  to  measure  in  general  and  office  output  per  hour 
can  only  be  measured  with  some  validity  in  industries  where  the 
percentage  of  office  workers  in  the  work  force  is  very  high  and  where 
output  measurement  problems  are  low.  (Panko,  1991:200) 

The  second  reason  for  the  problem  is  that  “while  IT  is  certainly  large  in 

relative  terms,  it  accounts  for  a  small  portion  of  all  office  spending,”  (Panko, 

1991:200). 

Another  review  of  corporate  performance  is  provided  in  the  article 
“Measuring  Corporate  Performance”  by  Brown  and  Laverick.  Their  conclusion 
was  that  standard  corporate  performance  measures  are  often  not  adequate  to 
describe  the  success  of  an  organization.  In  particular,  the  authors  discussed 
standard  accounting  measures  while  describing  the  shortcomings  of  each 
measure.  The  authors  noted  when  “serious  conflicts  arise  in  an  organization,  it 
is  inevitable  that  financial  considerations  override  all  other  alternatives”  (Brown 
and  Laverick,  1994:96).  Additionally,  the  authors  posited  that  “instead  of 
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yesterday’s  performance  measuring  yesterday’s  decisions,  what  are  needed  are 
measures  that  provide  today’s  decisions,  which  will  benefit  tomorrow’s 
performance”  (Brown  and  Laverick,  1994:96).  This  conclusion  directly  supports 
the  idea  that  we  must  find  a  new  model  that  better  describes  the  effect  that 
today’s  decisions  will  have  on  future  performance. 

Problem  Statement 

How  can  the  Air  Force  comply  with  the  ITMRA  with  respect  to: 

1 .  Paragraph  51 1 2  (c)  -  “include  explicit  criteria  for  analyzing  the  projected  and 
actual  costs,  benefits,  and  risks  associated  with  the  investments” 

2.  Paragraph  5122  (b)  5  -  “provide  for  identifying  for  a  proposed  investment 
quantifiable  measures  for  determining  the  net  benefits  and  risks  of  the 
investment” 

Research  Objectives 

To  solve  this  problem  there  are  three  steps  that  the  Air  Force  must  accomplish: 

1 .  Identify  possible  IT  investment  evaluation  models 

2.  Identify  which  models  meet  the  intent  of  the  ITMRA 

3.  Identify  which  models  are  feasible 
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Research  Questions 


The  objectives  listed  above  can  be  reduced  to  the  following  two  questions: 

1 .  What  models  are  currently  being  used  in  the  DoD  and  private  industry  to 
evaluate  their  investment  in  information  technology? 

2.  Of  these  models,  which  ones  can  the  Air  Force  use  to  evaluate  potential 
investment  in  IT  with  respect  to  the  ITMRA  and  with  regard  to  the  availability 
of  the  required  data? 

Scope 

The  research  for  this  thesis  will  have  a  broad  scope  by  necessity  to 
search  for  any  models  that  may  be  applicable  to  the  DoD.  This  search  will 
include  practitioner  journals,  academic  journals,  textbooks,  and  any  information 
available  through  the  internet. 

Management  Implications 

As  budgets  continue  to  fall,  or  at  least  stabilize,  the  Air  Force  must  receive 
more  value  from  its  IT  investments.  As  the  military  moves  towards  dramatically 
reducing  its  support  structure  to  invest  more  in  weapons  systems  and 
modernization,  IT  is  an  area  that  can  provide  some  of  the  productivity  and 
efficiency  gains  that  the  military  needs.  IT  is  one  of  the  largest  areas  of 
investment  in  support  equipment.  The  military,  and  government  in  general,  must 
ensure  that  its  investment  in  support  equipment  are  absolutely  the  most  it  can 
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receive  for  its  money.  By  properly  managing  the  investments  in  IT  the  Air  Force 
can  avoid  mistakes  of  the  past  such  as  the  DoD’s  problems  with  Ada  that  were 
alluded  to  earlier.  As  research  in  this  area  improves  and  better  techniques  are 
developed  for  IT  investment  valuation,  the  opportunity  to  get  the  most  from  our 
investments  will  improve. 

Overview 

Chapter  II  consists  of  a  review  of  the  pertinent  academic  and  practitioner 
literature  on  this  topic.  The  chapter  will  address  literature  that  describes  current 
DoD  policy,  productivity  concepts,  and  models  for  valuation  of  IT  investment. 
Chapter  II  will  also  describe  any  models  that  are  uncovered  by  the  research. 

Chapter  III  describes  the  methodology  for  this  research.  The 
methodology  is  based  on  Criterion-based  Congruence  Analysis,  Miles  and 
Huberman’s  qualitative  research  techniques.  Cooper  and  Emory’s  data  collection 
methods,  and  a  Delphi  group  for  variable  validation.  The  models  described  in 
Chapter  II  require  different  variables  to  quantify  IT  investment  valuation.  The 
concept  of  variable  validation  is  based  on  whether  or  not  the  data  for  these 
variables  is  available. 

Chapter  IV  will  describe  the  results  of  the  Delphi  panel,  which  will 
determine  the  models  that  may  be  useful  for  the  Air  Force. 

Chapter  V  contains  a  discussion  of  results  and  the  conclusions  regarding 
the  usefulness  of  the  models  based  on  the  variable  validation.  Chapter  V  will 


8 


close  with  a  discussion  of  the  implications  of  this  research  for  the  Air  Force,  and 
recommendations  for  future  research. 
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il.  Literature  Review 


Introduction 

This  literature  review  examines  the  current  body  of  knowledge  in  the  area 
of  IT  investment  valuation.  For  the  purpose  of  this  literature  review,  IT 
investment  will  be  defined  as  hardware,  software,  and  infrastructure  components 
such  as  routers,  hubs  and  cabling,  in  addition  to  other  expenses  related  to  IT 
such  as  training  and  maintenance.  This  review  will  focus  on  how  public  and 
private  sector  firms  evaluate  their  investments  in  IT  as  described  in  academic 
and  trade  literature. 

Types  of  Models 

In  one  effort  to  examine  IT  investment  valuation  models,  nine  objective 
and  eight  subjective  models  that  are  currently  used  by  firms  to  evaluate  their 
investments  were  examined  (Powell,  1992:30).  These  models  use  a  wide  range 
of  criteria  that  is  readily  available  to  most  organizations  and  apply  different 
techniques  such  questionnaires,  probabilities,  operational  performance 
parameters,  and  financial  data  analysis.  Powell  aggregated  the  views  of  many 
practitioners  in  the  field  and  stated  “most  are  of  one  opinion,  that  the  costs  and 
benefits  associated  with  computer  systems  are  difficult  to  quantify”  (Powell, 
1992:30). 
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After  brief  examination  of  the  models,  Powell  offered  that  “these  standard 
techniques  do  not  appear  to  be  widely  used,  even  though  they  have  been 
employed  in  other  fields  and  are  recognized  as  useful”  (Powell,  1992:40).  One 
minor  problem  with  Powell’s  research  was  that  several  of  the  concepts  were 
extended  series  of  guidelines  for  how  to  manage  IT  and  not  models  for 
assessing  its  value. 

The  next  section  discusses  the  models  discovered  during  the  research. 
The  models  are  classified  into  four  categories:  accounting  models  ,  options 
models,  economic  models,  and  miscellaneous  models.  These  categories  are 
simply  a  logical  presentation  of  different  models  recommended  by  IT  researchers 
and  practitioners. 

Accounting  Models.  The  first  category  is  the  Accounting  Models.  These 
models  are  the  basis  for  capital  investment  by  organizations.  They  include  such 
standard  concepts  as  Return  on  Investment  (ROI),  Payback  (PB),  and  Net 
Present  Value  (NPV).  Return  on  Investment  (ROI)  is  a  key  factor  used  by 
private  sector  firms  to  evaluate  any  capital  investment.  “Time  for  a  Fresh 
Approach  to  ROI”  describes  several  reasons  why  typical  ROI  approaches  are  not 
adequate  for  evaluating  IT  investments  (Moad,  1995:57).  ROI  doesn’t  account 
for  some  of  the  unique  capabilities  of  IT.  Moad  prescribed  a  model  under 
development  by  John  Henderson  of  Boston  University  based  on  Contingent 
Claims  Theory  (Moad,  1995:57).  This  model  takes  theory  from  options  pricing 
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and  suggests  a  new  way  to  evaluate  IT  that  provides  decision  makers  with  other 
ways  to  look  at  how  they  manage  IT. 

One  author  who  applied  standard  accounting  techniques  to  IT  investment 

valuation  was  Senn  in  his  textbook  Information  Systems  in  Management.  Senn 

gave  a  brief  description  of  Net  Present  Value  (NPV)  and  Payback  (PB),  (Senn, 

1978:523-524).  Even  in  this  early  examination  of  traditional  accounting 

methods,  Senn  noted  that  “the  intangible  benefits  are  ignored,  even  though  they 

may  be  the  most  important  ones  for  the  project”  (Senn,  1978:525).  The  model 

for  NPV  is  shown  in  equation  1  (Dos  Santos,  1991:74,  Senn,  1978:523). 

=  -Co  +  Z(t=1toT)^/(1+r)‘  (1) 

where  is  the  net  investment  cash  outflow,  T  is  the  project  life.  A,  is  the  cash 
inflow  at  time  period  t,  and  r  is  the  cost  of  capital 

The  model  for  Payback  is  shown  in  equation  2  (Senn,  1978:525). 

=  total  income  /  total  time  horizon  (2) 

Another  accounting  technique  is  Cost-Benefit  Analysis  (Kurnia  and 
Swatman,  1997).  Kurnia  and  Swatman  expanded  on  the  standard  cost  benefit 
analysis  to  specifically  address  areas  of  IT  investment.  One  particular  benefit  of 
this  work  was  the  authors’  adaptation  of  the  model  specifically  for  use  in  the 
public  sector  (Kurnia  and  Swatman,  1997:5).  The  model  uses  variables  specific 
to  IT  investment  that  are  readily  quantifiable.  The  first  year  model  is  as  follows: 
First  Year  (Kurnia  and  Swatman,  1997:5). 

Actual  Costs  =  Total  Costs  -  Total  Benefits  (3) 
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where  Total  Costs  =  Hardware  Costs  +  Software  Costs  +  Transmission  Costs 
+  Membership  Costs,  and  Total  Benefits  =  Processing  Cost  Savings  +  Paper 
Savings  +  Inventory  Savings  +  Quantifiable  Intangible  Benefits 

The  issues  relating  to  subsequent  year’s  investment  were  also  addressed. 
(Kurnia  and  Swatman,  1997:6).  The  subsequent  year  model  is  as  follows: 
Subsequent  Years  (Kurnia,  1997:6). 

Actual  Costs  =  On-going  Costs  -  On  going  Benefits  (4) 

where  On-going  Costs  =  Transmission  Costs  +  Software  Management  +  Fixed 
Annual  Costs,  and  On-going  Savings  =  Processing  Cost  Savings  +  Paper 
Savings  +  Inventory  Savings  +  Quantifiable  Intangible  Benefits 

One  of  the  first  researchers  to  notice  the  problems  with  traditional 
accounting  methods  in  evaluating  IT  investments  was  Peter  Lay  (1985:32).  After 
describing  Cost-benefit  Analysis  briefly.  Lay  examined  the  intangible  benefits 
that  IT  investment  provides.  In  particular  he  suggested  that  some  systems  “may 
provide  ONLY  intangible  benefits”  (Lay,  1985:32).  Lay  also  examined  other 
factors  where  traditional  evaluation  may  be  somewhat  misleading.  Many  costs 
of  IT  investment  cannot  be  accurately  quantified  before  initiation  of  the  project. 
Often  these  costs,  such  as  software  development,  are  often  inaccurately 
estimated.  With  the  speed  of  change  and  the  complexity  that  the  technology  of 
information  systems  changes,  it’s  easy  to  see  how  estimation  of  costs  on  a  major 
project  can  be  dramatically  different  than  originally  estimated.  Lay  points  out  that 
the  most  serious  drawback  of  cost/benefit  analysis  is  that  it  is  a  “rigid  tool  that 
cannot  bend  to  the  strategy  of  the  organization”  (Lay,  1985:35). 
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One  examination  of  valuation  techniques  compared  standard  financial 
methods  such  as  Discounted  Cash  Flow  (DCF)  and  NPV  to  an  options  valuation 
model  (Dos  Santos,  1991:71).  The  key  question  Dos  Santos  wanted  a  model  to 
answer  was  “Is  investment  in  the  new  technology  justified?”  (Dos  Santos, 
1991:72). 

After  a  brief  evaluation  of  the  financial  techniques,  Dos  Santos  concluded 
that  “traditional  financial  analysis  is  neither  recommended  nor  used  because  it 
does  not  adequately  deal  with  the  real  value  of  these  projects”  (Dos  Santos, 

1991 :72) .  This  conclusion  led  to  the  consideration  of  other  valuation  techniques 
that  may  be  able  to  better  quantify  the  unique  capabilities  of  IT.  A  model  that 
may  better  quantify  IT  investment  is  based  on  options  pricing  theory. 

Options  Models.  The  next  category  is  the  Options  Models.  Several 
researchers  have  discussed  using  options  valuation  techniques  as  a  way  to 
improve  the  IT  investment  process.  Moad  described  three  types  of  information 
that  the  Options  Model  adds  to  IT  Investment  analysis:  they  are  “current  and 
possible  future  business  strategies,  systems  capabilities  the  firm  may  want  in  the 
future,  and  risks  and  costs  of  various  IT  currently  used  by  the  firm”  (Moad, 
1995:58).  While  all  this  data  may  not  apply  specifically  to  DoD,  it  does  provide 
some  interesting  insights  into  how  better  to  evaluate  IT  investments.  Moad’s 
primary  finding  was  that  the  Options  Model  should  not  be  used  as  a  replacement 
for  ROI  but  can  add  more  valuable  information  to  the  decision  (Moad,  1995:58). 
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The  options  pricing  model  was  originally  developed  by  Margrabe  to  price 
an  option  to  exchange  one  risky  asset  for  another  (Margrabe,  1978:181).  Dos 
Santos  used  this  model  to  price  an  option  (a  decision)  “to  undertake  a  future 
project”  (Dos  Santos,  1991:80).  One  of  the  main  problems  with  this  model  is  the 
requirement  to  estimate  future  variables  required  by  the  model  such  as  the 
correlation  between  the  development  costs  and  revenues  for  the  second  stage 
project  and  the  variance  of  the  rate  of  change  of  development  costs  of  the 
second  stage  project  (Dos  Santos,  1991:82).  The  accuracy  of  these  estimates 
could  dramatically  effect  the  information  provided  to  the  decision  maker. 

This  model  places  value  on  managerial  flexibility  by  considering  the 
alternatives  that  may  be  available  in  the  future.  However,  Dos  Santos  closed 
this  article  by  noting  that  “methods  aimed  at  obtaining  accurate  estimates  of  the 
parameters  needed  to  use  this  model  will  need  to  be  developed”  (Dos  Santos, 
1991:87).  This  model  has  not  been  tested  extensively  in  organizations  but  has 
interesting  implications  as  further  research  is  conducted. 

Ram  Kumar  further  explored  the  applicability  of  options  pricing  models 
(Kumar,  1996:189).  Kumar  summarizes  two  major  approaches  to  options 
valuation,  the  Black-Scholes  model,  and  the  Margrabe  model.  The  research 
examined  one  example  and  concluded  that  option  valuation  techniques  would 
add  to  the  decision  if  “riskier  projects  increase  or  decrease  values  for  a  particular 
scenario”  (Kumar  1996:192).  The  model  for  Options  Pricing  is  shown  in  equation 
5  (Kumar,  1996:188,  Dos  Santos,  1991:81). 
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V  =  BiN(d,)-C,N(d2) 


(5) 


where  d^  =  (ln(Bi/Ci)  +  a^)  /  at^“'  dj  =  di  -  af V  =  Value  of  Option,  C  = 

expected  value  of  asset,  B  =  expected  return  on  asset,  t  =  time  to  expiration,  N  = 
standard  normal  density  function,  a  =  standard  deviations  of  rate  of  change 
between  B  and  C 

While  this  article  shows  a  solid  example  of  how  the  options  models  can 
contribute  to  IT  investment  decisions,  there  are  still  the  problems  described  in 
the  review  of  Dos  Santos’  work,  that  of  estimation  of  several  variables. 

Two  more  recent  articles  explored  the  idea  of  an  options  approach  to 
capital  investment.  These  articles  focus  on  capital  investment  in  assets  that  are 
easier  to  measure  than  IT,  however  some  of  the  principles  may  provide  insight 
into  how  to  use  this  method  to  better  quantify  IT  investment  (Dixit  and  Pindyck, 
1995:105).  Dixit  and  Pindyck  examined  the  problems  with  standard  approaches 
to  capital  investment  valuation.  The  authors  posited  that  the  standard  Net 
Present  Value  (NPV)  technique  does  not  include  all  pertinent  information  when 
making  a  decision  about  capital  investment.  Three  scenarios  were  examined,  to 
include  investment  in  oil  reserves,  electric  utilities,  and  commodity  price  volatility. 
In  these  three  scenarios,  the  options  technique  offered  decision  makers  a  way  to 
quantify  the  value  of  flexibility  in  delaying  decisions.  The  authors  concluded  that 
“options  create  flexibility,  and,  in  an  uncertain  world,  the  ability  to  value  and  to 
use  flexibility  is  crucial”  (Dixit  and  Pindyck,  1995:1 15). 
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Smith  and  Nau  also  examined  three  different  approaches:  risk-adjusted 
discount-rate  analysis,  option  pricing  analysis,  and  decision  analysis  (Smith  and 
Nau,  1995:795).  The  authors  used  a  capital  budgeting  example  to  show  how 
option  pricing  analysis  and  decision  analysis  can  be  integrated  to  improve  the 
information  for  the  decision  makers.  The  authors  concluded  that  option  pricing 
and  decision  analysis  are  compatible  in  their  results  and  can  be  profitably 
integrated  (Smith  and  Nau,  1995:812-813). 

In  another  attempt  to  add  an  additional  factor  to  IT  valuation,  Kumar 
addressed  how  options  can  be  used  to  quantify  the  flexibility  or  responsiveness 
that  IT  investment  gives  the  corporation  (Kumar,  1998:1).  IT  investment, 
because  of  its  applicability  to  many  different  functions,  gives  the  organization 
flexibility  to  change  focus  as  the  project  is  in  progress.  This  model  is  different 
from  earlier  work  by  Dos  Santos.  Dos  Santos  looked  specifically  at  the  value 
added  by  an  option  to  make  changes  to  the  project  by  a  specific  date.  In  this 
model,  Kumar  uses  a  model  originally  proposed  by  Majd  and  Pindyck  that  values 
sequential  investment  opportunities  but  does  not  force  the  subsequent 
investments  to  be  made  by  a  certain  date.  In  the  private  sector,  profits  and  ROI 
calculations  can  provide  some  insight  into  how  well  IT  is  managed.  However,  in 
government,  these  techniques  do  not  apply  well  because  of  their  reliance  on 
income  as  a  variable.  Productivity  may  be  one  method  for  evaluating  IT  for 
government  use. 
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Economic  Models.  The  third  category  is  the  Economic  Models.  These 
models  are  based  in  economic  theory  such  as  consumer  surplus  and  the 
production  function.  While  relying  heavily  on  economic  theory,  the  models 
presented  in  this  paper  have  been  initially  tested  by  the  original  authors. 

Brynjolfsson,  who  has  researched  extensively  in  the  field  of  IT 
productivity,  reviewed  the  paradox  of  why  productivity  has  not  been  shown  to 
have  grown  while  there  have  been  significant  investments  in  IT  over  the  past  30 
years  (Brynjolfsson,  1993:68).  Brynjolfsson  then  suggested  several  reasons  for 
this  phenomena.  The  first  is  that  organizations  do  not  measure  the  correct 
outputs  and  inputs.  The  second  is  the  lags  in  learning  that  are  inherent  in  the 
move  to  new  systems.  Next  is  the  redistribution  and  dissipation  of  profits. 
Finally,  the  author  suggested  that  there  is  significant  mismanagement  of 
information  and  technology  (Brynjolfsson,  1993:73). 

Since  the  government  cannot  use  profitability  measures  to  guide  IT 
investment,  productivity  measures  would  be  one  area  where  it  could  quantify 
savings.  However,  as  noted  by  Brynjolfsson,  researchers  do  not  yet  have  a 
strong  grasp  on  the  relationship  between  IT  investment  and  increased 
productivity  (Brynjolfsson,  193:76). 

Wurth  discussed  the  standard  definition  of  productivity  as  the  change  in 
outputs  divided  by  the  change  in  inputs  (Wurth,  1993:233).  This  definition  is 
standard  in  economic  theory  and  provides  the  basis  from  which  this  research 
must  start.  The  author  noted  an  important  concept  that  future  research  must 
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consider:  the  concept  that  significant  gains  are  normally  achieved  when  there  are 
qualitative  change  in  processes  and  not  quantitative  changes  in  inputs. 

One  of  the  key  articles  in  IT  investment  theory  looks  at  three  different 
measures  of  IT  investment;  Productivity,  Profitability,  and  Consumer  Surplus  (Hitt 
and  Brynjolfsson,  1996:121).  The  authors  offer  an  interesting  perspective  by 
considering  productivity  and  consumer  surplus  in  addition  to  the  more  common 
measure,  profitability. 

Consumer  surplus  provides  interesting  implications  for  use  with 
government.  Consumer  surplus  is  a  concept  of  value  that  quantifies  the  value 
passed  on  to  consumers  from  investments  in  IT.  Within  the  government, 
consumers  could  be  considered  as  internal  and  external  customers.  The  model 
for  Consumer  Surplus  is  shown  in  equation  6  (Hitt  and  Brynjolfsson,  1996:135). 

Surplus =  Va  (S H1+  S  t )  *  log  (p ,/  p ,*i)  *  V  (6) 

where  t  =  time  period,  S  =  IT  Stock  to  Value  Added,  P  =  Price  of  IT  Stock, 
V=Value  Added 

Productivity  is  also  a  measure  that  can  be  applied  more  easily  to 
government  organizations.  The  model  (the  Cobb-Douglas  production  function) 
employed  by  Hitt  and  Brynjolfsson  had  variables  that  considered  IT  stock,  non¬ 
computer  capital,  and  labor.  The  regression  equation  used  to  estimate  the  Value 
Added  showed  extremely  positive  results  across  all  tests.  The  model  for  the 
Production  Function  Approach  is  shown  in  equation  7  (Hitt  and  Brynjolfsson, 
1996:130). 
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V  =  exp  (EtD,  +  Sj.i  Dj)C 


(7) 


where  C  =  Total  IT  Stock,  K  =  Non-Computer  Capital,  L  =  Labor,  V  =  Value 
Added,  D  =  Dummy  variable  to  control  for  year,  p  =  output  elasticity  of  IT  stock 
(%  change  in  output  / 1  %  increase  in  IT  Stock) 

Profitability  has  its  foundations  in  the  common  measures  of  Return  on 
Assets  (ROA),  Return  on  Investment  (ROI),  and  Shareholder  Return.  These 
areas  provide  key  measures  of  how  well  a  business  uses  its  capital,  however, 
these  measures  are  not  easily  applied  to  nonprofit  government  organizations. 
The  model  for  Business  Profitability  Analysis  is  shown  in  equation  8  (Hitt  and 
Brynjolfsson,  1996:132). 

Profitability  Ratio  =  ao  +  *  (ITRate)  +  control  variables  +  s  (8) 

where  ITRate  =  ratio  of  IT  stock  to  firm  employees,  a  is  a  constant 

In  addition  to  the  work  of  Brynjolfsson  and  Hitt,  others  have  applied  pure 
economic  theory  to  IT  investment.  Bakos  and  Kemerer  looked  at  different 
economic  theories  and  how  they  might  be  applied  to  IT  valuation  (Bakos  and 
Kemerer,  1992:365).  The  authors  first  described  the  economic  characteristics  of 
IT.  Then,  the  authors  discussed  several  economic  theories  describing  the 
current  applications  and  then  offered  ideas  for  future  research  in  the  specific 
areas.  The  authors  main  conclusion  was  that  “the  benefits  are  present,  but 
simply  not  being  measured  correctly  with  current  approaches”  (Bakos  and 
Kemerer,  1992:380).  The  economic  models  described  above  examine  different 
areas  where  IT  investment  effects  an  organization.  Other  models,  such  as  those 
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discussed  in  the  next  section,  explore  what  effects  IT  has  on  the  management  of 
the  organization. 

Miscellaneous  Models.  The  final  category  is  the  Miscellaneous  Models. 
These  models  are  often  more  subjective  in  nature  and  at  some  point  may  rely  on 
subjective  measures  of  performance. 

Strassmann  suggested  “one  area  to  address  in  order  to  increase 
productivity  is  information  management”  (Strassmann,  1998a).  Information 
management  can  be  defined  as  the  cost  of  “all  information  processes,  not  just 
those  which  involve  IT”  (Strassmann,  1998a).  The  model  for  the  Cost  of 
Information  Management  is  shown  in  equation  9  (Strassmann,  1998a). 

=  Cost  of  Sales,  General  and  Administrative  (9) 

+  Cost  of  Research  and  Development 

Strassmann  found  that  “over  80%  of  US  organizations  Invest  more  on 
information  than  on  capital”  (Strassmann,  1998a).  The  model  for  the  Cost  of 
Capital  is  shown  in  equation  10  (Strassmann,  1998a). 

=  (Shareholder  Equity  +  Capital  Surplus)  *  Interest  Rate  (10) 

Information  Productivity  is  another  area  that  Strassmann  examined.  “The 
exploration  of  information  productivity  is  necessary  because  even  a  favorable 
evaluation  of  computer  expenditures  offers  no  guarantee  of  competitive  viability” 
(Strassmann,  1997:xviii).  The  model  for  Information  Productivity,  Private  Sector 
is  shown  in  equation  11  (Strassmann,  1998a). 
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=  Value  Added  by  Information  (1 1 ) 

/  Cost  of  Information  Management 

To  calculate  the  value  of  Information  Productivity,  an  organization  must 
first  calculate  the  Value  Added  by  Information.  To  calculate  the  value  added 
Strassmann  suggests  that  “capital  is  a  commodity,  and  deducts  the  cost  of 
paying  rent  on  it”  (Strassmann,  1998a).  The  model  for  Value  Added  by 
Information  is  shown  in  equation  12  (Strassmann,  1998a). 

=  Net  Profit  -  (Financial  Capital  Assets  (12) 

*  Interest  Rate  for  Borrowing) 

Strassmann  also  developed  a  model  of  Information  Productivity  for  the 
public  sector  which  “doesn’t  take  account  of  revenue  and  capital  in  the  same 
way”  (Strassmann,  1998a).  The  model  for  Information  Productivity,  Public 
Sector  is  shown  in  equation  13  (Strassmann,  1998a). 

=  Cost  of  Operations  (1 3) 

/  Cost  of  Information  Management 

Strassmann  stated  that  there  is  “no  direct  link  between  IT  spending  and 
productivity”  (Strassmann,  1998a).  “A  CIO  needs  to  be  able  to  compare  IT 
expenditure  with  that  of  competitors... it  is  also  useful  to  compare  divisions  within 
an  organization”  (Strassmann,  1998a).  The  model  for  IT  Spending  is  shown  in 
equation  14  (Strassmann,  1998a). 

=  a  +  b*(Cost  of  Sales,  General,  and  Administrative)  (14) 

+  c*Profits  +  D*Desktops  +  e*Professionals  -  f*Executives 
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Strassmann  stated  that  “the  wealth  of  an  organization  is  based  on  its 
accumulation  of  useful  knowledge  -  its  knowledge  capital”  (Strassmann,  1998a). 
The  model  for  Knowledge  Capital  is  shown  in  equation  15  (Strassmann,  1998a). 

=  Value  Added  by  Information  (15) 

/  Interest  Rate  for  Equity  Capital 

Strassmann  posited  that  “for  all  practical  purposes,  the  existing  approach 
to  valuing  IT  assets  is  useless  except  as  a  way  of  complying  with  regulatory  and 
taxation  purposes”  (Strassmann,  1997:337).  Information  is  different  from  other 
assets  and  should  be  managed  differently.  One  key  suggestion  for  successful  IT 
management  is  to  “include  the  residual  value  of  a  project  in  all  project  payoff 
calculations”  (Strassmann,  1997:335).  The  model  for  the  Residual  Value 
Formula  is  shown  in  equation  16  (Strassmann,  1997:337). 

=  Change  in  Information  Technology  Assets  =  (equipment  acquisition  (16) 

-  equipment  depreciation)  +  (development  acquisition 

-  development  depreciation)  +  (software  acquisition 

-  software  depreciation)  +  (training  acquisition  -  training  depreciation) 

Another  attempt  to  quantify  Knowledge  Capital  lead  to  replacement  of  The 
Value  Added  by  Information  (equation  12, 15)  model  with  the  Management 
Value  Added  model  (Strassmann,  1998b).  Strassmann  defines  Management 
Value  Added  as  “what  is  left  over  after  absolutely  all  costs  are  fully  accounted 
for”  (Strassmann,  1998b).  The  revised  model  for  Knowledge  Capital  is  shown  in 
equation  17  (Strassmann,  1998b). 
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=  Management  Value  Added 
/  Price  of  Capital 


(17) 


In  1992,  Robert  Kaplan  and  David  Norton  developed  a  method  called  the 
Balanced  Scorecard  to  present  executives  with  information  in  a  manner  that 
directly  relates  the  information  to  specific  goals  within  the  company  in  four  areas: 
the  financial  perspective,  the  customer  perspective,  the  internal  business 
perspective,  and  the  innovation  and  learning  perspective  (Kaplan  and  Norton, 
1992:72).  This  technique  was  developed  after  a  “year  long  research  project  with 
12  companies  at  the  leading  edge  of  performance  measurement”  (Kaplan  and 
Norton,  1992:71).  The  Balanced  Scorecard  is  designed  to  put  “strategy,  not 
control”  at  the  center  of  performance  management  (Kaplan  and  Norton  1992:79). 
The  authors  posited  that  this  new  approach  is  consistent  with  many  current 
management  trends  including  team  focus,  cross-functional  integration,  and 
customer-supplier  partnerships.  The  Balanced  Scorecard  was  tested  in  a  later 
article  on  specific  companies  including  Apple  computer  and  Advanced  Micro 
Devices  (Kaplan  and  Norton,  1993:140-141).  In  a  final  article,  Kaplan  and 
Norton  further  refined  the  use  of  this  technique  for  businesses  (Kaplan  and 
Norton,  1996:75).  The  key  point  in  this  article  was  how  companies  could  use  the 
Balanced  Scorecard  to  align  management  processes  and  focus  the  entire 
organization  on  implementing  its  long-term  strategy  (Kaplan  and  Norton, 
1996:85). 
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Also  in  1996,  Kaplan  and  Norton  published  a  book,  The  Balanced 
Scorecard,  that  further  explained  the  applications  of  their  model,  in  particular, 
how  public  organizations  could  apply  the  financial  measures  (Kaplan  and  Norton, 
1996: 179).  One  of  the  key  problems  with  IT  investment  valuation  in  public 
organizations  is  how  to  consider  the  budget.  Kaplan  and  Norton  note  that 
meeting  a  budget  is  not  a  success  if  the  customer  loses  service  or  if  the  mission 
of  the  agency  is  compromised.  The  authors  note  that  tangible  objectives  must 
be  defined  for  customers  and  that  financial  considerations  can  play  an  enabling 
or  constraining  role  but  will  rarely  be  the  primary  objective  (Kaplan  and  Norton, 
1996:180).  The  balanced  scorecard  is  a  technique  that  focuses  on  managing  a 
large  business,  so  how  does  the  balanced  scorecard  fit  in  with  IT  valuation? 

Van  Grembergen  and  Van  Bruggen  applied  the  balanced  scorecard  to  the 
valuation  of  IT  investments  (Van  Grembergen  and  Van  Bruggen,  1998:1).  The 
authors  changed  the  basic  areas  of  the  Balanced  Scorecard,  the  financial 
perspective,  the  customer  perspective,  the  internal  business  perspective,  and  the 
innovation  and  learning  perspective,  into  areas  specific  to  the  IT  function  within  a 
business:  user  orientation,  corporate  contribution,  operational  excellence,  and 
future  orientation  (Van  Grembergen  and  Van  Bruggen,  1998:2).  These  areas 
are  further  broken  down  in  to  specific  mission  and  objective  statements. 
Additionally,  the  authors  suggested  outcome  and  performance  measures  for 
each  area.  The  majority  of  these  measures  are  quantifiable  so  that  they  could 
be  tracked  in  relation  to  goal  attainment. 
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Van  Grembergen  and  Van  Bruggen  proposed  this  technique,  composed 
of  elements  of  “information  economics  and  business  reengineering”  (Van 
Grembergen  and  Van  Bruggen,  1998:7),  as  a  model  that  addresses  both  readily 
quantifiable  variables  and  more  abstract  measures  of  user  satisfaction  as  an 
overall  management  tool  for  IT.  The  authors  have  not  tested  this  model  \with  any 
organizations. 

The  Balanced  Scorecard  is  shown  in  Tabies  1  through  5  (Van 
Grembergen  and  Van  Bruggen,  1998:9-11).  Table  1  shows  the  Balanced 
Scorecard  for  IT.  This  table  focuses  on  the  key  contributions  of  IT  for 
organizations.  Table  1  is  divided  into  four  key  areas:  User  Orientation, 

Corporate  Contribution,  Operational  Excellence,  and  Future  Orientation  (Van 
Grembergen  and  Van  Bruggen,  1998).  These  four  areas  are  further  reduced 
into  their  respective  mission  statements  and  specific  objectives. 
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Table  1.  The  Balanced  Scorecard 


User  Orientation 

How  do  the  users  view  the 

IT  Department? 

Corporate  Contribution 

How  does  management  view  the 

IT  Department? 

Mission 

To  be  the  preferred  supplier  of 
information  systems  and  to  exploit 
business  opportunities  maximally  through 
information  technology 

Objectives 

•  preferred  supplier  of  applications 

•  preferred  supplier  of  operations 

•  partnership  with  users 

•  user-satisfaction 

Mission 

To  obtain  a  reasonable  business 
contribution  of  investments  in  IT 

Objectives 

•  control  of  IT  expenses 

•  sell  IT  products  and  services  to  third 
parties 

•  business  value  of  new  IT  projects 

•  business  value  of  IT  function 

Operational  Excellence 

How  effective  and  efficient  are  the 

IT  processes? 

Future  Orientation 

Is  IT  positioned  to  meet  future 
challenges? 

Mission 

Efficiently  deliver  IT  products  and 
services 

Objectives 

•  efficient  software  development 

•  efficient  operations 

•  acquisition  of  PCs  and  PC-software 

•  problem  management 

•  training  users 

•  management  of  IT  personnel 

•  use  of  communications  software 

Mission 

Develop  opportunities  to  answer  future 
challenges 

Objectives 

•  permanent  training  and  education  of  IT 
personnel 

•  expertise  of  IT  personnel 

•  age  of  the  applications  software 

•  research  into  emerging  information 
technologies 

Table  2  shows  the  specific  variables  for  the  subcomponet  of  Corporate 
Contribution.  The  four  proposed  measures  are;  Control  IT  Expenses,  Sell  to 
Third  Parties,  Business  Value  of  New  IT  Projects,  and  Business  Value  of  IT 
Function  (Van  Grembergen  and  Van  Bruggen,  1998). 


27 


Table  2.  Measures  for  Corporate  Contribution 


Control  IT  Expenses  _ 

•  percentage  above  or  within  IT  budget 

•  allocation  of  the  different  budget  items 

•  IT  budget  as  a  percentage  of  turnover 

•  IT  expenses  per  staff  member  _ 

Sell  to  Third  Parties _ 

•  financial  benefits  stemming  from  selling  products  and  services _ 

Business  Value  of  the  New  IT  Projects _ 

•  financial  evaluation  based  on  ROI,  NPV,  IRR,  PB 

•  business  evaluation  based  on  Information  Economics _ 

Business  Value  of  the  IT  Function  _ 

•  percentage  of  the  development  capacity  engaged  in  strategic  projects 

•  relationship  between  new  developments/infrastructure  investments/replacement 

investments  _ _ 


Table  3  shows  the  specific  variables  for  the  subcomponent  of  User 
Orientation.  The  three  proposed  measures  are:  Preferred  IT  supplier, 
Partnership  With  Users,  and  User  Satisfaction  (Van  Grembergen  and  Van 
Bruggen,  1998). 
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Table  3.  Measures  for  User  Orientation 


Preferred  IT  Supplier _ 

•  %  of  applications  managed  by  IT 

•  %  of  applications  delivered  by  IT 

•  %  of  in-house  applications _ 

Partnership  With  Users _ _ 

•  index  of  user  involvement  in  generating  new  strategic  applications 

•  index  of  user  involvement  in  developing  new  applications 

•  frequency  of  IT  Steering  Committee  meetings _ 

User  Satisfaction _ 

•  index  of  user  friendliness  of  applications 

•  index  of  user  satisfaction 

•  index  of  availability  of  applications  and  systems 

•  index  of  functionality  of  applications 

•  %  of  application  development  and  operations  within  the  Service  Level  Agreement 

(SLA) _ 


Table  4  shows  the  variables  for  Measures  for  Operational  Excellence. 
The  seven  proposed  measures  are:  Efficient  Software  Development,  Efficient 
Operations,  Acquisition  PC’s  and  PC  Software,  Problem  Management,  User 
Education,  Managing  IT  Staff,  and  Use  of  Communications  Software  (Van 
Grembergen  and  Van  Bruggen,  1998). 


Table  4.  Measures  for  Operational  Excellence 

Efficient  Software  Development _ 

•  %  of  changes  and  adjustments  made  throughout  different  development  stages 

•  number  of  defects  per  function  point  in  the  first  year  of  production 

•  number  of  function  points  per  person-per  month 

•  average  number  of  days  late  in  delivering  software 

•  average  unexpected  budget  increase 

•  %  of  projects  performed  within  SLA 

•  %  of  code  that  is  reused 

•  %  of  maintenance  activities 

•  visible  and  invisible  backlog  _ 

Efficient  Operations _ 

•  %  unavailability  of  the  mainframe 


%  unavailability  of  the  network 
response  times  per  category  of  users 
%  of  jobs  done  within  set  times 
%  of  reruns 

average  time  between  systems  failures 


Problem  Management _ _ _ 

•  average  answer  time  of  help  desk 

•  %  of  questions  answered  with  set  time 

•  %  of  solutions  with  SLA _ 

User  Education _ 

•  %  of  users  that  already  received  education  (per  technology/application) 

•  quality  index  of  education _ 

Managing  IT  Staff  _ 

•  number  of  people  hours  that  can  be  charged  internally  or  externally 

•  %  of  people  hours  that  are  charged  on  projects 

•  satisfaction  index  of  IT  staff _ 

Use  of  Communications  Software _ 

•  %  of  IT  staff  that  can  access  groupware  facilities  (inter-  and  intranet) 

•  %  of  IT  staff  that  effectively  use  groupware  facilities _ 


Table  5  shows  the  variables  for  Measures  for  Future  Orientation.  The  four 
proposed  measures  are:  Permanent  Education  of  Staff,  Expertise  of  IT  Staff, 
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Age  of  Applications  Portfolio,  and  Research  Into  Emerging  Technologies  (Van 
Grembergen  and  Van  Bruggen,  1998). 


Table  5.  Measures  for  Future  Orientation 

Permanent  Education  of  Staff _ 

•  number  of  educational  days  per  person 

•  education  budget  as  a  %  of  total  budget _ 

Expertise  of  IT  staff _ 

•  number  of  years  of  IT  experience  per  staff  member 

•  age  pyramid  of  the  IT  staff _ 

Age  of  the  Applications  Portfolio _ 

•  number  of  applications  per  category 

•  number  of  applications  younger  than  five  years _ 

Research  Into  Emerging  Technologies _ 

•  %  of  budget  spent  on  IT  research _ 


The  Federal  Government  has  also  contracted  for  and  published  three 
guides  to  improve  the  IT  investment  process.  These  publications  are  similar  in 
nature  and  all  provide  general  guidance  for  IT  investment  decisions.  However, 
they  cannot  be  considered  models  because  of  the  very  general  nature  of  their 
scope.  There  are  no  brief  sections  that  could  be  excerpted  and  used  as  a  model 
to  evaluate  IT  investment.  Overall,  the  guides  do  provide  an  excellent  overview 
of  how  to  manage  the  IT  investment  process  and  are  useful  as  an  initiation  point. 
The  guides  are  as  follows: 

1.  Air  Force  Information  Technology  Investment  Performance  Measurement 
Guide  (ANDRULIS  Corporation,  1997). 
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2.  Assessing  Risks  and  Returns:  A  Guide  for  Evaluating  Federal  Agencies  IT 
Investment  Decision  Making  (GAO,  1 997). 

3.  Department  of  Defense  Guide  for  Managing  Information  Technology  as  an 
Investment  and  Measuring  Performance  (Vector  Research  Incorporated,  1997). 


Summary 

There  are  many  models  that  both  academics  and  practitioners  have 
examined  in  their  search  to  adequately  quantify  IT  investment.  None  of  these 
models  is  universally  accepted  and  the  debate  continues  on  how  to  analyze 
these  investments.  Public  sector  organizations  also  lack  cash  flows  to  use  in  the 
evaluation  process.  The  models  discussed  are  summarized  in  Table  6  below. 
Table  6  shows  the  equation  number  as  listed  in  Chapter  II,  the  Author’s  name, 
the  name  of  the  model,  and  the  equation  for  the  model. 


Table  6.  Summary  of  Models 


Equation 

Number 

Author 

Name 

Equation 

1 

Senn,  Dos  Santos 

Net  Present  Value 

=  -Co  +  I(t=1toT)A./ 

(1+r)‘ 

2 

Senn 

Payback 

=  total  income  /  total  time 
horizon 

3 

Kurnia  and  Swatman 

Actual  Costs 

Total  Costs  -  Total  Benefits 

4 

Kurnia  and  Swatman 

Actual  Costs 
(Subsequent  Years) 

On-going  Costs  -  On-going 
Benefits 

5 

Kumar,  Dos  Santos 

Options 

V  =  B,N(d,)-C,N(d,) 

6 

Hitt  and  Brynjolfsson 

Consumer  Surplus 

1/2(S,.,+  S,)*log(p,/p„,) 
*V 

7 

Hitt  and  Brynjolfsson 

Production  Function 

V  =  exp(i:,Dt  +  i:j.i  Dj) 

Q  p1  P2  L  p3 
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8 

Hitt  and  Brynjolfsson 

Profitability 

=  Oq  +  a,  *  (ITRate)  + 
control  variables  +  s 

9 

Strassmann 

Cost  of  Information 
Management 

=  Cost  of  Sales,  General 
and  Administrative  + 

Cost  of  Research  and 
Development 

10 

Strassmann 

Cost  of  Capital 

=  (Shareholder  Equity  + 
Capital  Surplus)  *  Interest 
Rate 

11 

Strassmann 

Information 

Productivity 

Value  Added  by  Information 
/  Cost  of  Information 
Management 

12 

Strassmann 

Value  Added  By 
Information 

=  Net  Profit  -  (Financial 
Capital  Assets  *  Interest 

Rate  for  Borrowing) 

13 

Strassmann 

Information 

Productivity 

Cost  of  Operations  /  Cost  of 
Information  Management 

14 

Strassmann 

IT  Spending 

=  a  +  b*(Cost  of  Sales, 
General,  and 

Administrative)  +  c*Profits  + 
□‘Desktops  + 
e*Professionals  - 
^Executives 

15 

Strassmann 

Knowledge  Capital 

Management  Value  Added  / 
Interest  Rate  for  Equity 
Capital 

16 

Strassmann 

Residual  Value 

=  Change  in  Information 
Technology  Assets  = 
(equipment  acquisition  - 
equipment  depreciation) 

+  (development  acquisition  - 
development  depreciation) 

+  (software  acquisition 
-  software  depreciation) 

+  (training  acquisition  - 
training  depreciation) 

17 

Strassmann 

Knowledge  Capital 

Management  Value  Added  / 
Price  of  Capital 

18 

Van  Grembergen  and 
Van  Bruggen 

Balanced  Scorecard 

See  Table  1  through  5 
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III.  Methodology 


Introduction 

This  study  is  an  exploratory  investigation  of  the  models  currently  in  use  or 
under  development  that  may  help  answer  the  research  questions.  Before  the 
research  design  is  discussed  at  length,  consider  again  the  research  questions. 

1.  What  models  are  currently  being  used  in  the  DoD  and  private  industry 
to  evaluate  their  investment  in  information  technology? 

2.  Of  these  models,  which  ones  can  the  Air  Force  use  to  evaluate 
potential  investment  in  IT  with  regard  to  the  availability  of  the  required 
data? 

Both  qualitative  and  quantitative  techniques  are  applicable  although 
exploration  relies  more  heavily  on  qualitative  techniques  (Cooper  and  Emory, 
1995:1 18).  The  research  design  for  this  study  is  a  qualitative  technique. 

Research  Design 

Criterion-based  Congruence  Analysis  is  a  technique  developed  by  Peifer 
to  select,  screen,  and  analyze  qualitative  data  (Peifer,  1997:107).  A  graphical 
representation  of  this  model  is  shown  in  Figure  1.  This  model  is  designed  to  use 
filters  to  select  the  objects  to  be  analyzed.  Embedded  in  the  Criterion-based 
Congruence  Analysis  model  is  the  work  by  Miles  and  Huberman  (1994:12)  and 
their  view  of  qualitative  analysis. 
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Figure  1.  Criterion-Based  Congruence  Analysis  (Peifer,  1997:111) 


Miles  and  Huberman  suggest  a  view  of  qualitative  analysis  based  on  four 
concepts:  Data  Collection,  Data  Reduction,  Data  Display  and  Conclusion 
DrawingA/erification.  During  the  Criterion-based  Congruence  Analysis,  all 
models  were  analyzed  continuously.  Miles  and  Huberman  stated  that 
“qualitative  data  analysis  is  a  continuous,  iterative  enterprise”  (Miles  and 
Huberman,  1994:12).  Miles  and  Huberman  suggested  the  following  technique 
for  analysis:  “Data  analysis  can  be  defined  as  three  concurrent  flows  of  activity: 
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data  reduction,  data  display,  and  conclusion  drawing/verification”  (Miles  and 
Huberman,  1994:12). 

Figure  2  shows  a  graphical  representation  of  the  relationship  between  these 
four  concepts  as  originally  outlined. 


Figure  2.  Quantitative  Data  Analysis  (Miles  and  Huberman,  1994:12) 

Next,  the  four  concepts  of  qualitative  analysis  as  described  by  Miles  and 
Huberman  and  shown  in  Figure  2,  are  discussed  in  detail  along  with  the  specific 
techniques  used  to  accomplish  each  task. 

Data  Collection 

Data  Collection  occurs  during  a  “sustained  period”  (Miles  and  Huberman, 
1994:10)  and  allows  continuous  analysis  of  the  qualitative  data.  For  collection  of 
data.  Cooper  and  Emory  described  the  technique  of  document  analysis  (Cooper 
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and  Emory,  1995: 11 9).  This  technique  was  used  to  discover  what  techniques 
practitioners  and  academia  are  currently  using  and  prescribing. 

Cooper  and  Emory  stated  that  secondary  data  sources  can  be  classified 
into  external  and  internal  sources  (Cooper  and  Emory,  1995:241).  For  the 
purpose  of  this  investigation,  external  sources  are  those  outside  the  DoD  or 
government.  These  data  sources  could  come  from  almost  anywhere.  For 
example,  this  research  will  investigate  sources  from  academia  and  private 
industry,  and  therefore  opens  up  the  research  to  sources  such  as  academic  and 
practitioner  journals.  The  second  type  of  secondary  data  source  is  internal 
sources.  These  sources  are  from  entities  within  the  federal  government  or  the 
DoD  and  can  be  found  at  such  sources  as  Defense  Technical  Information  Center 
(DTIC)  and  on  the  internet  at  government  sites. 

There  are  many  people  involved  in  research  to  identify  practical  models 
for  government  and  industry  to  properly  evaluate  their  investment  in  information 
technology.  These  people  can  be  classified  into  two  categories:  academia  and 
practitioners.  For  the  purpose  of  this  research,  consider  practitioners  as  those 
people  and  organizations  within  the  private  sector  of  the  economy  whose  work  is 
based  on  maximizing  efficiency  or  profits. 

Literature  from  practitioners  and  academia  are  at  the  opposite  ends  of  the 
spectrum  with  regard  to  how  they  share  their  information.  First,  literature  from 
academia  is  readily  available  due  to  the  nature  of  the  researchers.  For  the 
researchers  to  get  recognition,  they  must  publish  their  models.  However, 
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literature  from  practitioners  is  somewhat  more  difficult  to  obtain.  If  a  company 
has  models  that  it  can  use  to  be  more  efficient,  it  is  unlikely  the  company  will 
publicize  these  models  so  that  their  competitors  could  use  them  to  increase  their 
efficiency. 

Government  entities  are  somewhat  different  than  academia  and 
practitioners.  Government  entities  are  relatively  open  with  their  information. 
However,  what  makes  that  government  different  is  the  fact  that  there  are 
organizations  within  government  that  could  be  classified  as  research  based  or 
practitioners.  For  instance,  the  Government  Accounting  Office  (GAO)  publishes 
many  guides  on  efficiency  and  other  topics,  but  does  use  them  because  of  the 
nature  of  their  work.  The  GAO  is  an  overseer  on  government  efficiency.  There 
are  also  practitioners  within  government  such  as  the  DoD  who  will  publish 
information  on  how  to  complete  certain  tasks.  Often  this  information  may  be 
usable  in  other  organizations.  These  organizations  are  in  a  similar  situation  to 
those  in  academia.  It  is  beneficial  for  them  to  get  recognized  for  their 
contributions  to  increasing  government  efficiency. 

The  goal  of  the  search  of  external  data  sources  is  to  identify  any  models 
that  are  relevant  to  the  research  question.  The  search  was  conducted  in  two 
specific  domains:  the  facilities  available  from  libraries  and  the  internet.  The 
search  was  limited  to  those  publications  in  the  information  technology  realm.  An 
initial  review  of  several  economic  and  financial  journals  showed  some  articles 
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that  addressed  the  topic,  however,  their  models  were  still  very  much  in  the 
theoretical  stages  of  development. 

The  two  libraries  searched  are  located  at  the  Air  Force  Institute  of 
Technology  and  Wright  State  University.  The  search  mechanisms  at  the  libraries 
were  the  Business  Periodicals  Guide  and  the  electronic  library  search  system 
called  EBSCO  from  EBSCO  Publishing  (for  results  from  EBSCO  search,  see 
Table  1). 

EBSCO  is  a  system  that  transitions  from  the  standard  library  search  to  a 
computerized  search  focusing  on  academic  and  practitioner  journals.  It  is  a 
somewhat  more  narrowly  focused  system  than  the  standard  internet  search 
engines.  However,  EBSCO  is  a  computerized  search  system  that  follows  the 
standard  computerized  search  technique  described  below. 

Cooper  and  Emory  (1995:250),  suggested  the  following  outline  for  a 
computerized  search: 

1.  Select  an  appropriate  database.  To  initiate  the  data  collection,  the  following 
meta  search  engines  were  examined:  Yahoo,  Alta  Vista,  and  Hot  Bot. 

2.  Type  in  the  search  terms.  These  search  engines  were  used  with  the  key 
words  starting  with  “information  technology”  and  limited  by  four  additional  words: 
investment,  valuation,  evaluation,  and  model.  Advanced  search  techniques 
were  used  to  limit  the  amount  of  hits.  For  example  in  all  the  search  engines,  it  is 
possible  to  limit  the  results  to  sites  that  match  all  the  key  words  by  using  the 
“and"  or  the  “+”  symbol.  For  example,  the  first  search  was  initiated  by  the  key 
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words  “information  and  technology  and  investment.”  The  remaining  searches 
were  completed  by  changing  “investment”  to  “valuation”  to  “evaluation”  and 
finally  to  ’’model.” 

3.  Examine  the  number  of  possible  citations  -  expand  or  limit  that  number.  As 
shown  in  Table  1,  the  search  engines  provided  the  number  of  citations  stated  in 
the  second  column.  When  examining  the  number  of  citations,  it  would  appear 
from  the  numbers  that  the  search  engine  that  Hot  Bot  used  is  not  as  restrictive 
as  the  other  three  search  engines  when  using  the  “and”  criteria. 

As  stated  earlier,  every  search  was  initiated  with  the  two  key  words 
“information”  and  “technology.”  Then  the  additional  key  words,  shown  in  Table  7 
below  and  discussed  in  paragraph  2  above,  were  added  to  limit  the  search. 
Table  7  shows  in  the  search  engine  used,  each  key  word  added  to  “information” 
and  “technology,”  and  the  number  of  hits  from  each  source. 


40 


Table  7.  Results  of  Computerized  Search  Engines 


Search 

Engine 

Key  Word 

Number  of  Hits 

Yahoo 

evaluation 

9 

investment 

54 

valuation 

4 

model 

6 

Alta  Vista 

evaluation 

55 

investment 

872 

valuation 

125 

model 

15 

Hot  Bot 

evaluation 

323,444 

investment 

301,883 

valuation 

260,090 

model 

449,191 

EBSCO 

evaluation 

55 

investment 

90 

valuation 

2 

model 

37 

One  problem  with  these  searches  was  the  type  of  citations  the  search 
engines  found.  The  oven/vhelming  majority  of  the  citations  were  references  to 
consulting  companies  and  referred  mainly  to  the  services  they  provided.  The 
second  largest  category  of  citations  was  for  college  courses  providing 
information  on  the  course  and  in  some  cases  the  instructor.  At  this  point  a  more 
narrowly  focused  search  was  required. 

To  accomplish  this,  web  sites  specifically  relating  to  information 
technology  were  examined.  These  sites  were  found  using  keywords  “information 
technology”  followed  by  an  investigation  of  the  citations.  In  addition  to  the  sites 
found  in  this  search,  sites  from  other  sources  were  investigated.  Due  to  the 
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nature  of  the  Information  Research  Management  academic  program  at  AFIT, 
many  sites  addressing  various  topics  relating  to  information  technology  have 
been  discussed  in  class,  and  informally  among  the  students.  Table  8  is  a  listing 
of  the  specific  IT  sites  examined  that  had  content  specifically  relating  to  IT  policy. 


Table  8.  Information  Technology  Internet  Sites 


Title 

Web  Address 

CIO’s  Executive  Research  Center 

http://www.cio.com/ 

Electronic  College  of  Process  Innovation 

http://w\Artw.dtic.mil/c3i/bprcd/ 

IS  Effectiveness 

http://theweb.badm.sc.edu/grover/ 

isworld/ 

IS  World 

http://wvm.isworld.org/isworld .  htm  1 

IT  Policy  OnRamp 

http://www.itpolicy.gsa.gov/index.htm 

Business  Researchers  Interests 

http://vmw.brint.com/interest.html 

Signal 

http://wvm.us.net/signal/ 

Air  Force  Communications  Agency 

http://vwm.afca.scott.af.mil/pa/ 

Government  Accounting  Office 

http://vwm.gao.gov 

Air  Force  CIO 

http://vvvm.cio.hq.af.mil/ 

Navy 

http://vwm.doncio.navy.mil/ 

Army 

http://vwm.gordon.army.mil/ 

4.  Retrieve  an  initial  citation  or  citations.  The  meta  search  engines  were 
adequately  restricted  by  the  use  of  the  keywords.  This  allowed  research  of  all 
hits  on  every  site  except  Hot  Bot  where  the  combination  keywords  did  not  seem 
to  filter  the  citations.  The  more  specific  sites  listed  in  Table  8  were  reviewed 
exhaustively  to  discover  any  models  they  may  reference.  Any  link  that  remotely 
suggested  a  link  to  a  model  or  discussion  of  models  pertinent  to  this  research 
was  investigated. 
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5.  Review  the  citations  and  decide  if  the  search  needs  modification  for 
additional,  fewer  or  more  precise  citations.  After  the  refined  review  in  step  four, 
the  search  needed  no  additional  modification.  Models  were  available  to  examine 
as  a  starting  point  for  this  research 

6.  Investigate  the  citations.  The  citations  were  reviewed  and  subjected  to  the 
following  steps  in  the  Criterion-based  Congruence  Analysis  model. 

Data  Reduction 

Data  Reduction  “refers  to  the  process  of  selecting,  focusing,  simplifying, 
abstracting,  and  transforming  the  raw  data  that  appear  in  written-up  field  notes” 
(Miles  and  Huberman,  1994:10).  For  this  study,  data  reduction  will  focus  on 
identifying  and  selecting  models  that  may  provide  better  information  to  DoD  and 
government  decision  makers. 

Model  Filtering.  The  models  will  first  be  filtered  based  on  their 
relevance  to  the  problem.  The  key  criteria  for  any  model  is  relevance  (Christ, 

1 966:4).  For  the  model  to  have  value  for  this  research,  it  must  address  how  the 
Air  Force  can  better  evaluate  proposed  IT  investment  as  required  by  the  ITMRA. 
Figure  3  shows  different  areas  of  IT  evaluation  where  models  may  focus.  By 
definition,  this  research  is  focused  in  the  area  of  “Organizational  Impact”  (Delone 
and  McLean,  1998). 
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Figure  3.  Computer  System  Effectiveness  (Delone  and  McLean,  1998) 

A  Delphi  panel,  fully  described  below,  was  used  to  increase  the  validity  of 
this  research.  In  the  final  step.  Conclusion  Drawing  and  Verification,  the  Delphi 
panel’s  opinions  on  the  relevance  and  availability  of  data  for  the  variables  used 
by  the  various  models  will  be  aggregated  to  evaluate  the  potential  usefulness  of 
the  models  for  the  Air  Force. 

The  Delphi  technique,  “the  name  for  a  set  of  procedures  for  eliciting  and 
refining  the  opinions  of  a  group  of  people”  (Dalkey,  1967:1),  was  used  for  data 
reduction  and  conclusion  drawing  in  the  Criterion-based  Congruence  Model. 

The  Delphi  technique  is  used  because  it  offers  “anonymity,  controlled  feedback 
and  statistical  group  response”  (Dalkey,  1967:3).  This  allows  the  participants  to 
respond  freely  with  no  external  forces  to  consider. 

The  researcher  made  an  initial  decision  based  on  his  experience  in 
communications  on  whether  or  not  the  information  described  by  each  variable  is 
available,  not-available,  collectable,  or  not  collectable. 

The  Delphi  panel  was  then  asked  to  validate  the  researcher’s  conclusions 
based  on  their  experience  in  the  Air  Force.  To  gather  the  Delphi  group’s 
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responses,  a  copy  of  Appendix  A,  Appendix  B,  and  Table  10  were  sent  to  each 
Major  Command  (MAJCOM)  Communications  Directorate.  Table  9  shows  the 
MAJCOMs  that  were  offered  the  opportunity  to  participate  in  the  Delphi  panel. 
The  first  column  is  the  list  of  MAJCOMs  solicited  and  the  second  column  shows 
whether  or  not  the  MAJCOM  was  able  to  respond. 


Table  9.  Major  Command  Participation 


Major  Command 

Participated  (Y  /  N) 

ACC 

Yes 

AETC 

No 

AFMC 

No 

AFSOC 

Yes 

AMC 

Yes 

PACAF 

No 

SPACECOM 

No 

USAFE 

Yes 

Appendix  A  is  the  electronic  mail  cover  letter.  People  were  selected  to 
form  the  Delphi  panel  based  on  the  information  provided  in  the  instructions 
(Appendix  A).  The  Delphi  panel  was  asked  to  concur  or  non-concur  with  the 
researcher’s  initial  conclusions.  Appendix  B  is  an  extended  definition  of  each 
variable.  Table  10  is  complete  list  of  variables  compromising  the  models. 
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Based  on  the  responses  from  the  Delphi  panel,  the  models  are  filtered  on 
the  basis  of  whether  or  not  the  data  to  input  into  the  variables  is  available,  not 
available,  collectable  or  not  collectable.  This  matrix  shows  the  variables  in  the 
first  column  with  a  brief  definition.  The  next  column,  “available,”  is  defined  as 
whether  the  data  is  currently  used  for  any  other  reason  such  as  in  quality 
indicators.  The  next  column,  “not-available,”  is  for  data  that  is  not  available  for 
any  reason.  One  example  would  be  if  a  variable  required  a  data  point  for  “Profit.” 
Government  agencies  are  non-profit  and  do  not  collect  this  sort  of  data  or 
generate  profit.  Some  agencies  are  fee  for  service  and  therefore  do  generate 
revenue.  For  this  reason,  models  requesting  cash  flows  will  not  be  discarded. 
The  next  column  is  “collectable.”  Collectable  is  defined  as  data  that  may  not  be 
currently  used  but  could  be  relatively  easily  and  inexpensively  collected.  The 
fourth  column  is  “not-collectable.”  This  column  is  for  variables  that  would  be  too 
expensive  or  time  consuming  to  obtain.  This  issue  will  not  be  discussed  in  this 
research  due  to  its  dynamic  nature.  Different  organizations  operate  in  diverse 
environments  that  may  cause  a  variable  that  is  collectable  in  one  instance  to  be 
prohibitively  expense  to  collect  in  another. 

This  results  from  the  Delphi  panel  will  be  presented  in  Chapter  IV  in  Table 
1 1 .  This  table  shows  what  percentage  of  the  Delphi  panel  concurred  with  the 
initial  categorization  of  the  variables.  When  there  is  disagreement  among  the 
Delphi  panel  about  in  which  category  the  variable  should  be  in,  the  percentage  of 
the  Delphi  panel  members  that  voted  for  each  category  is  shown. 
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Table  10.  Model  Variables  and  Mapping 


-  expectea  return  on 
asset 

'  c  -  expected 
investment  in  asset 

'  t  -  time  to  expiration  of 
choice 

‘  P  -  correlation 
coefficient  between 
return  on  asset  and 
return  investment  in 
asset 

■  (j  - standard  deviation 
of  return  on  asset  and 
return  investment  in 
asset 


return 

8  p  -  price  of  IT  stock 
~9“  V  -  valuelidded 
s  -  IT  stock  /  value 
added 

%  above  or  within 
budget 

^2  allocation  of  different 
budget  items 
IT  budget  as  a  %  of 
turnover 

IT  expenses  per  staff 
member 

financial  benefits  from 
selling  to  third  parties 
financial  evaluation 
16  based  on  ROl,  NPV, 
IRR 

%  of  business 
development  capacity 
^  engaged  In  strategic 
projects 


b  OT  applications 
managed  by  IT 
.Q  %  of  applications 
delivered  by  IT 
2q  %  of  in-house 
applications 


equire 
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Table  10.  Model  Variables  and  Mapping  (continued) 


e 

I  Available 

jN 

ot-Available 

r 

Collectable 

rrequency  or  1 1 
21  Steering  Committee 
Meetings 
%  of  applIcatioTi 
development  and 
operations  within  the 
SLA 

%  of  changes  and 
adjustments  made 
throughout  different 
development  stages 


number  of  defects  per 

24  function  point  in  first 
year  of  production 
number  of  function 

25  points  per  person  per 
month 

average  number  of 

26  days  late  in  delivering 
software 

average  unexpected 
budget  Increase 
%  of  projects  performed 
within  SLA 

29  %  of  code  reused 

%  of  maintenance 

30 

activities 

%  unavailability  of 
mainframe 

%  unavailability  of  the 
network 

response  times  per 
category  of  users 
%  of  jobs  completed 
within  set  times 
35  %  of  reruns 

average  time  between 
systems  failures 
ratio  of  operational 
costs  to  installed  MIPS 
average  lead  time  for 
deliveries 

average  answer  time  of 
help  desk 
%  of  questions 
40  answered  within  set 
time 


ot-Collectable 
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Table  10.  Model  Variables  and  Mapping  (continued) 


ot-Available  I  Collectable  I  Not-Collectable  |  Maps  to  Mode 


b  OT  solutions  witnin 


%  of  users  that  already 
^  received  education  per 
technology/application 
quality  index  of 
education 

number  of  hours  that 
44  can  be  charged 
Internally  or  externally 
.  %  ot  people  hours 
^  charged  on  projects 
satisfaction  Index  ot  II  “ 
staff 

- %of  IT  staff  that  can 

47  access  groupware 
facilities 


7o  OT  1 1  Starr  mat  can 
48  effectively  use 
groupware  facilities 
number  of  education 
days  per  person 


education  budget  as  a 
percentage  of  IT  budget 

"number  of  years 
experience  per  staff 
member 

■  age  pyramid  of  IT  staff 

■  number  of  applications 
per  age  category 

'  number  of  applications 
younger  than  5  years 

■  %  of  budget  spent  on 
research 

■  hardware  costs 

■  software  costs 
transmission  costs 

■  processing  costs 
savings 

■  paper  savings 

■  inventory  savings 

■  quantifiable  intangible” 
benefits 

■  software  maintenance 
costs 

■  fixed  annual  costs 
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Table  10.  Model  Variables  and  Mapping  (continued) 


Available 

Not-Available 

Collectable  |  Not-Collectable 

Maps  to  Model 

X 

9,14 

Cost  of  Research  and 
Development 

X 

9.10 

67 

Shareholder  Equity 

X 

10 

68 

Capital  Surplus 

X 

10 

69 

Management  Value 
Added 

X 

17 

70 

Costs  of  Management 

X 

71 

Costs  of  Inform^Fon 
Management 

X 

13 

72 

Value  Added  by 
Information 

X 

11 

73 

Information  Productivity 

X 

16 

74 

Equipment  Depreciation 

X 

16 

75 

Development 

Depreciation 

X 

16 

76 

Software  Depreciation 

X 

16 

77 

Training  Depreciation 

X 

16 

Data  Display 

Data  Display  is  defined  “as  an  organized  assembly  of  information  that 
permits  conclusion  drawing  and  action  taking”  (Miles  and  Huberman,  1994:11). 
After  the  models  are  identified  as  possibly  being  useful  in  some  situation 
(naturally  not  all  models  will  be  useful  in  all  circumstances).  Many  of  these 
models  took  dramatically  different  approaches  to  examining  IT  investment  and 
focused  on  different  areas.  Remember  that  the  focus  of  this  study  is  to  discover 
models  and  review  their  usefulness. 

The  data  display  exists  in  Chapter  IV.  Chapter  II  describes  the  models 
and  the  variables  they  require.  This  information  is  summarized  in  a  matrix  format 
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for  ease  of  viewing  and  evaluating.  “The  most  frequent  form  of  display  for 
qualitative  data  in  the  past  has  been  extended  text”  (Miles  and  Huberman, 

1994:1 1).  However,  Miles  and  Huberman  later  state  this  type  of  display  “is 
dispersed,  sequential  rather  than  simultaneous,  poorly  structured,  and  extremely 
bulky”  (Miles  and  Huberman,  1994:1 1).  To  alleviate  this  problem  a  matrix 
structure  consisting  of  the  identified  variables  in  columns  and  the  filters  in  rows 
will  be  used.  The  variables  are  separated  from  the  models  to  allow  independent 
review  of  the  variable  without  any  bias  towards  what  problem  the  model  may 
address.  This  matrix  is  designed  to  demonstrate  why  any  variable  and 
eventually  any  model  was  filtered. 

Conclusion  Drawing  and  Verification 

Conclusion  Drawing  and  Verification  “is  beginning  to  decide  what  things 
mean,  is  noting  regularities,  patterns,  explanations,  possible  configurations, 
causal  flows,  and  propositions”  (Miles  and  Huberman,  1994:11).  The  basis  for 
the  conclusions  is  the  results  of  the  Delphi  panel  in  Chapter  IV.  Models  that  do 
not  have  a  majority  opinion  from  the  Delphi  panel  that  the  required  variables  are 
available  or  collectable  will  be  classified  as  not  currently  useful  for  the  Air  Force. 
The  final  conclusions  based  on  this  filtering  will  be  fully  discussed  in  Chapter  V. 

An  important  part  of  research  is  understanding  what  you  have  “not”  done 
in  addition  to  what  you  have.  Any  gaps  that  this  research  uncovers  will  be 
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discussed  at  length  in  Chapter  V  where  recommendations  for  future  research  are 
made. 

Summary 

This  is  an  exploratory  study  designed  to  select,  screen,  and  analyze 
models  that  will  improve  the  Air  Force’s  IT  investment  decision  process.  For  this, 
the  Criterion-based  Congruence  Model  was  chosen  for  its  structured  approach 
and  its  combination  of  previously  recognized  techniques  such  as  those 
prescribed  by  Miles  and  Huberman,  and  Cooper  and  Emory.  A  Delphi  panel 
then  validated  the  researcher’s  initial  conclusions  based  on  their  experience  in 
Air  Force  communications. 
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IV.  Analysis 


Results 

This  chapter  will  describe  the  results  of  the  Criterion-Based  Congruence 
Analysis  model  described  in  Chapter  III.  Chapter  I  stated  the  research  objectives 
as  follows: 

1 .  Identify  possible  IT  investment  evaluation  models 

2.  Identify  which  models  meet  the  intent  of  the  ITMRA 

3.  Identify  which  models  are  feasible  for  the  Air  Force 

These  objectives  were  met  by  specific  parts  of  the  Criterion-Based 
Congruence  Analysis  and  are  discussed  in  order.  IT  investment  evaluation 
models  discovered  during  the  data  collection  stage  were  summarized  in  Table  6 
in  Chapter  II. 

Section  5122  (b)  5  tasked  government  agencies  to  “provide  for  identifying 
for  a  proposed  investment  quantifiable  measures  for  determining  the  net  benefits 
and  risks  of  the  investment”  (United  States  Congress,  Sec  5122  (b)  5).  The 
models  all  meet  the  intent  of  this  law.  Each  model  uses  quantifiable  measures  to 
examine  the  benefits  and  risks  of  proposed  IT  investment. 

The  models  examine,  in  a  variety  of  ways,  the  costs  and  benefits  of  IT 
investment.  Quantifying  the  risks  of  an  investment  is  a  somewhat  more  abstract 
idea.  However,  by  efficiently  evaluating  the  costs  and  benefits  of  a  proposed  IT 
investment,  the  Air  Force  is  reducing  the  risk  of  the  purchase.  Scenario  analysis 
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and  sensitivity  analysis  are  two  of  the  key  principles  of  risk  management  (Cooley 
and  Roden,  1988:400).  Scenario  analysis  is  defined  as  “analysis  of  a  capital 
project’s  profitability  under  three  sets  of  assumptions:  pessimistic,  most  likely, 
and  optimistic”  (Cooley  and  Roden,  1988:383).  Scenario  analysis  is  defined  as 
“analysis  of  the  effects  that  key  input  variables,  changed  one  at  a  time,  have  on 
the  profitability  of  a  capital  investment”  (Cooley  and  Roden,  1988:5).  Many  of 
the  models  do  not  have  profitability  as  a  specific  objective,  however,  the 
objective  of  the  model  can  be  analyzed  using  the  same  principles  of  scenario 
and  sensitivity  analysis.  By  using  models  that  require  specific  identification  of 
variables,  the  Air  Force  has  the  basic  information  required  to  complete  the 
required  risk  analysis. 

For  models  to  be  useful  to  the  Air  Force,  the  required  variables  must  be 
available.  Table  1 1  is  a  copy  of  Table  10  with  the  exception  of  the  final  column. 
The  final  column,  Delphi,  shows  whether  the  members  of  the  panel  concurred  or 
non-concurred  with  the  researcher’s  initial  classification  of  the  variables.  In  the 
event  of  non-concurrence,  the  percentage  of  members  of  the  Delphi  panel  voting 
for  each  category  is  shown. 
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Table  11.  Delphi  Panel  Results 


Available  Not-Available  Collectable  Not-Collectable 


-  expectea  return  on 
asset 

“  C  -  expected 
investment  in  asset 
T^time  fo  expiration  of 
choice 


P  -  correlation 
coefficient  between 
return  on  asset  and 
return  investment  in 
asset 


a-stanaar 
of  return  on  asset  and 
return  investment  in 
asset 


period 

^  r  -  risk  free  rate  of 
'  return 

8  p  -  price  of  IT  stock 


-  value  aaoei 
slock /value 
added 

%  above  or  within 
'  budget 

allocation  of  different 
^  budget  items 
^  IT  budget  as  a  %  of 
^  turnover 

|-|- expenses  per“sl^“ 
14  r 
member 

financial  benefits  from 
^  selling  to  third  parties 
financial  evaluation 
16  based  on  ROI,  NPV, 
IRR 

o/q  business 
.  development  capacity 
'  engaged  in  strategic 
projects 

%  of  applications 
^  managed  by  IT 
%  of  applications 
delivered  by  IT 
%  of  in-hduse 
^  applications 
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Table  10.  Delphi  Panel  Results  (continued) 


Variable 

Available 

Not-Available 

Collectable 

Not-Collectable 

21 

frequency  of  IT  Steering 
Committee  Meetings 

X 

25% 

non-concur 

22 

%  of  application 
development  and 
operations  within  the 
SLA 

X 

concur 

23 

%  of  changes  and 
adjustments  made 
throughout  different 
development  stages 

X 

concur 

24 

number  of  defects  per 
function  point  in  first 
year  of  production 

X 

25% 

non-concur 

25 

number  of  function 
points  per  person  per 
month 

X 

concur 

26 

average  number  of 
days  late  in  delivering 
software 

X 

concur 

27 

average  unexpected 
budget  increase 

25% 

X 

non-concur 

28 

%  of  projects 
performed  within  SLA 

X 

concur 

29 

%  Of  code  reused 

X 

concur 

30 

%  of  maintenance 
activities 

X 

concur 

31 

%  unavailability  of 
mainframe 

X 

25% 

non-concur 

32 

%  unavailability  of  the 
network 

X 

25% 

non-concur 

33 

response  times  per 
category  of  users 

25% 

X 

non-concur 

%  of  jobs  completed 
within  set  times 

X 

concur 

35 

%  of  reruns 

25% 

X 

non-concur 

36 

average  time  between 
systems  failures 

X 

25% 

non-concur 

37 

ratio  of  operational 
costs  to  installed  MIPS 

25% 

X 

non-concur 

38 

average  lead  time  tor 
deliveries 

X 

concur 

1 - 

39 

average  answer  time  of 
help  desk 

25% 

25% 

X 

non-concur 

40 

%  of  questions 
answered  within  set 
time 

25% 

X 

non-concur 

Table  10.  Delphi  Panel  Results  (continued) 


Variable 


b  oT  solutions  witnin 


%  of  users  that  already 
received  education  per 
technology/application 


education 

num  ber  of  hours  that 
44  can  be  charged 
internally  or  externally 
%  of  people  hours 
^  charged  on  projects 
satisfaction  index  of  IT 
staff 

%of  IT  staff  that  can 

47  access  groupware 
facilities 

%  of  IT  staff  that  can 

48  effectively  use 
groupware  facilities 


numoer  oteaucation 
^  days  per  person 

50  education  budget  as  a 
percentage  of  IT  budget 
number  of  years 

51  experience  per  staff 
member 

~ST  age  pyramid  of  IT  staff 
number  of  applications 
per  age  category 
number  of  applications 
younger  than  5  years 
7o  of  budget  spent  on  IT 
^  research 

56  hardware  costs 


transmission  costs 


processing  costs 
savings 
'  paper  savings 
'  inventory  savings 
■quantifiable  intangible 
benefits 


software  maintenance 
costs 

■  fixed^nnu^“cdsts 


Available  Not-Available  Collectable  Not-Collectable 
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Table  10.  Delphi  Panel  Results  (continued) 


Variable 

Available 

Not-Available 

Collectable 

Not-Collectable 

65 

Cost  of  Sales,  (general 
and  Administrative 

25% 

X 

25% 

25% 

nonconcur 

66 

Cost  of  Research  and”” 
Development 

. 

X 

concur 

67 

Shareholder  Equity 

X 

concur 

BE" 

Capital  Surplus 

X 

concur 

69 

Management  Value 
Added 

X 

concur 

7tr 

Costs  of  Management 

25% 

X 

non-concur 

71 

Costs  of  Information 
Management 

75% 

X 

non-concur 

72 

Value  Added  by 
Information 

X 

25% 

non-concur 

73 

Information  Productivity 

X 

concur 

74 

Equipment 

Depreciation 

X 

concur 

75 

Development 

Depreciation 

25% 

non-concur 

76  i 

Software  Depreciation” 

25% 

X 

non-concur 

77 1 

Training  Depreciation 

25% 

X 

non-concur 

Findings 

If  any  model  is  purely  equation  based  and  cannot  be  used  without  all 
relevant  variables,  it  is  not  valid  for  Air  Force  use  at  this  point  in  time.  For 
example,  if  a  model  used  a  variable  in  multiplication  or  division  and  the  variable 
was  not  available  or  not  collectable,  the  whole  model  could  not  be  used. 
However,  in  addition  and  subtraction  models,  all  variables  need  not  be  available 
or  collectable.  In  this  type  of  model,  the  variable  can  be  reduced  to  zero  and  will 
not  effect  the  other  terms  in  the  model.  There  may  be  changes  in  the  future  that 
may  change  this  initial  conclusion.  For  example,  if  an  organization  went 
completely  fee  for  service  or  had  a  system  that  was  dedicated  to  fee  for  service 
operations,  a  model  that  is  not  currently  valid  may  be  useful  to  set  prices  or 
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make  decisions  among  systems.  Following  is  a  discussion  of  the  results  of  the 
Delphi  panel  with  regard  to  each  model. 

The  NPV  model  (equation  1)  currently  has  limited  value  for  the  Air  Force. 
The  Delphi  panel  unanimously  concurred  that  Variable  6,  cash  flow  at  end  of 
period,  was  not  available.  This  variable  is  critical  for  use  of  this  model. 

The  Payback  model  (equation  2)  suffers  from  the  same  limitations  as  the 
NPV  model.  This  model  requires  a  cash  flow,  Variable  6,  to  compute  the  length 
of  time  until  the  initial  investment  is  returned. 

The  Cost-Benefit  Analysis  (equation  3  and  4)  is  the  first  model  that  may 
be  useful  for  the  Air  Force.  There  was  significant  non-concurrence  from  the 
Delphi  panel  on  the  collectability  and  availability  of  the  variables.  The  researcher 
initially  concluded  that  Variables  58-60  and  61  were  not-collectable.  However, 
the  majority  of  the  Delphi  panel  disagreed  and  concluded  that  these  variables 
were  in  fact  collectable  or  already  available.  Remember,  this  model  was 
developed  specifically  with  regard  to  the  public  sector  (Kurnia  and  Swatman, 
1997:5). 

The  Options  model  (equation  5)  has  also  shows  limited  usefulness  for  the 
Air  Force  at  the  present  time.  The  Delphi  panel  unanimously  concurred  that 
Variable  5,  standard  deviation  of  return  on  asset  and  investment  in  asset,  was 
not  available.  This  model  also  had  mixed  results  regarding  Variable  1 ,  expected 
return  on  asset,  and  Variable  4,  correlation  coefficient.  Fifty  percent  and 
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seventy-five  percent  respectively,  of  the  Delphi  panel  concurred  that  these 
variables  are  not  available. 

The  Consumer  Surplus  model  (equation  6)  is  not  useful  for  the  Air  Force 
at  the  present  time  because  of  Variable  9,  Value  Added  (mapped  from  output 
elasticity,  P).  The  Delphi  panel  unanimously  concurred  that  Variable  9  is  not 
collectable. 

The  Production  Function  model  (equation  7)  is  not  useful  to  the  Air  Force. 
The  Delphi  panel  unanimously  concurred  that  Variable  9,  Value  Added,  was  not 
collectable. 

The  Profitability  model  (equation  8)  may  be  useful  for  the  Air  Force  in  very 
specific  circumstances.  The  Delphi  panel  agreed  that  Variable  8  is  collectable. 
However,  this  model  requires  constants  based  on  other  factors  to  predict 
profitability.  In  situations  where  the  system  is  to  be  used  on  a  fee  for  service 
situation,  the  model  may  be  useful  to  estimate  a  price  high  enough  to  cover  the 
costs  described  by  Variable  8,  Price  of  IT  Stock. 

The  Cost  of  Information  Management  model  (equation  9)  also  had  mixed 
results  within  the  panel.  The  Delphi  panel  was  split  on  Variable  65,  Cost  of 
Sales,  General,  and  Administrative.  Fifty  percent  of  the  panel  voted  that 
Variable  65  was  available  or  collectable  and  fifty  percent  voted  that  the  variable 
was  not  available  or  not  collectable.  The  second  variable  in  this  equation  was 
Variable  66,  Cost  of  Research  and  Development.  The  Delphi  panel  unanimously 
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agreed  that  this  variable  was  not  collectable.  However,  if  this  variable  was  set  a 
zero,  it  would  not  make  the  model  invalid. 

The  Cost  of  Capital  model  (equation  10)  is  not  useful  to  the  Air  Force. 

This  model  requires  measures  of  Variable  67,  Shareholder  Equity,  and  Variable 
68,  Capital  Surplus,  that  the  Delphi  panel  unanimously  concurred  were  not 
available. 

The  Information  Productivity  model  (equation  11)  is  based  on  other 
models  in  this  research.  This  model  is  the  result  of  equation  12  divided  by 
equation  9.  While  the  results  of  the  Delphi  panel  on  equation  9  were  mixed,  on 
equation  12  they  were  not.  As  described  in  the  next  paragraph,  equation  12  is 
not  useful  due  to  its  requirement  for  net-profit. 

The  Value  Added  by  Information  (equation  12)  model  is  not  useful  to  the 
Air  Force.  This  model  includes  Variable  6,  Cash  flow,  which  the  Delphi  panel 
unanimously  concurred  was  not  collectable. 

The  Information  Productivity,  Public  Sector  model  (equation  13)  is 
dependent  on  equation  9,  the  Cost  of  Information  Management.  If  an 
organization  can  use  equation  9,  then  it  also  will  be  able  to  use  this  model  as 
another  measure. 

The  IT  Spending  model  (equation  14)  is  currently  useful  to  the  Air  Force  in 
certain  situations.  This  model  is  designed  to  compare  IT  spending  among 
organizations  and  could  be  used  to  quantify  how  well  lower  level  organizations 
are  using  their  IT  budget.  The  one  variable  that  is  not  available  Variable  6,  Cash 
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flow,  (which  ultimately  could  be  reduced  to  profits)  could  be  reduced  to  zero  and 
have  no  net  effect  on  the  value  when  used  to  compare  similar  organizations. 

The  Knowledge  Capital  model  (equation  15)  is  not  currently  useful  to  the 
Air  Force.  This  model  relies  on  a  value  from  equation  12,  which  the  Delphi  panel 
unanimously  concurred  was  not  available. 

The  Residual  Value  model  (equation  16)  may  be  useful  to  the  Air  Force  in 
certain  situations.  A  majority  of  the  Delphi  panel  determined  that  Variable  76, 
Software  Depreciation,  and  Variable  77,  training  Depreciation,  was  available.  A 
majority  of  the  Delphi  panel  concurred  that  Variable  75,  Development 
Depreciation,  was  not  available.  However,  as  in  the  IT  spending  model 
(equation  14),  if  this  variable  were  reduced  to  zero,  there  would  be  no  effect 
when  comparing  similar  organizations.  This  model  could  be  used  (similarly  to 
equation  14)  to  compare  management  practices  among  similar  organizations 
within  the  Air  Force. 

The  Knowledge  Capital  model  (equation  17)  is  not  currently  useful  for  the 
Air  Force.  This  model  also  relies  on  input  from  equation  12,  which  is  not 
available. 

The  final  model,  the  Balanced  Scorecard  (model  18)  is  slightly  different 
from  the  other  models.  Instead  of  looking  at  quantifiable  measures  and  using 
them  in  an  equation  to  generate  a  specific  number,  the  Balanced  Scorecard 
seeks  to  use  quantifiable  measures  to  judge  how  an  organization  is  meeting  it 
goals.  Of  the  44  variables  in  the  Balanced  Scorecard  (variables  1 1-55),  the  vast 
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majority  of  the  variables  were  either  available  or  collectable.  A  majority  of  the 
Delphi  panel  concluded  that  Variable  15,  Financial  Benefits  of  Selling  to  Third 
Parties,  was  not  available.  Variable  33,  Response  Times  per  Category  of  Users, 
Variable  35,  Percentage  of  Reruns,  Variable  37,  Ratio  of  Operational  Costs  to 
Installed  MIPS,  Variable  44,  Number  of  Hours  that  can  be  Charged  Internally  or 
Externally,  and  Variable  45,  Percentage  of  People  Hours  Charged  on  Projects, 
all  received  a  majority  of  Delphi  panel  concurrence  that  they  were  not 
collectable.  If  these  six  variables  were  not  used,  the  Balanced  Scorecard  would 
still  have  38  variables  that  could  be  used  to  measure  an  organization’s 
performance  in  relation  to  specific  goals  or  in  relation  to  another  organization. 

Table  12  is  a  summary  of  the  discussion  of  the  usefulness  of  the  models. 
The  first  column  is  the  equation  number  from  Chapter  II.  The  second  column  is 
the  name  of  the  equation.  The  third  and  fourth  columns  represent  whether  or  not 
the  data  is  available  or  collectable  and  whether  or  not  the  data  is  not  available  or 
not  collectable. 
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Table  12.  Summary  of  Models 


Equation 

Number 

Equation  Name 

Available  / 
Collectable 

Not-Available  / 
Not-Collectable 

1 

Net  Present  Value 

X 

2 

Payback 

X 

3 

Actual  Costs 
(First  Year) 

X 

4 

Actual  Costs 
(Subsequent  Years) 

X 

5 

Options 

X 

6 

Consumer  Surplus 

X 

7 

Production  Function 

X 

8 

Buiness  Profitability 

X 

9 

Cost  of  Information 
Management 

X 

10 

Cost  of  Captial 

X 

11 

Information 
Productivity  (Private 
Sector) 

X 

12 

Value  Added  By 
Information 

X 

13 

Information 
Productivity  (Public 
Sector) 

X,  dependant  on 
Equation  9 

14 

IT  Spending 

X 

15 

Knowledge  Capital 

X 

16 

Residual  Value 

X 

17 

Knowledge  Capital 
(Revised) 

X 

18 

Balanced  Scorecard 

X 
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V.  Conclusions 


Overview 

The  ITMRA  has  tasked  government  organizations  to  develop  and  use 
quantitative  methods  to  evaluate  potential  IT  investment  decisions.  This 
research  set  out  with  the  objective  of  discovering  what  IT  investment  evaluation 
models  were  currently  in  use  both  in  public  or  private  sector  organizations.  The 
second  objective  was  to  identify  those  models  which  met  the  intent  of  the 
ITMRA,  specifically  to  “identify  net  benefits  and  risks  of  proposed  investments” 
(United  States  Congress,  1996:  Section  5122  (b)  5).  The  final  objective  was  to 
determine  which  of  those  models  were  feasible  for  Air  Force  use. 

A  total  of  18  models  were  found  ranging  from  basic  accounting  models  to 
more  theoretical  econometric  models.  The  full  list  of  these  models  is  shown  is 
Table  6  in  Chapter  II.  These  models  had  a  total  of  77  variables,  many  of  which 
were  common  to  several  models.  The  complete  variable  list  is  shown  in  Table 
10  in  Chapter  III. 

The  Delphi  panel  then  screened  the  variables  to  determine  which  models 
were  usable  given  the  information  required  for  the  variables  composing  a  specific 
model.  This  research  suggests  that  there  are  models  available  that  may  improve 
the  Air  Force’s  decision  making  process  in  the  area  of  IT  investment. 
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Conclusions 


There  are  five  models  that  may  help  the  Air  Force  better  manage  the 
benefits  and  risks  of  IT  investment:  Cost  Benefit  Analysis,  Residual  Value,  IT 
Spending,  Cost  of  Information  Management,  and  the  Balanced  Scorecard.  In 
addition,  two  other  models  may  be  useful  in  certain  situations.  The  Business 
Profitability  and  Information  Productivity  models  use  data  that  may  be  collectable 
in  certain  organizations 

The  Cost  Benefit  Analysis  model  shows  how  the  Air  Force  can  examine 
an  IT  investment  in  terms  of  initial  investment  and  subsequent  management. 

The  one  possible  problem  with  this  model  is  the  variable  “Quantifiable  Intangible 
Benefits.”  As  stated  earlier  in  the  literature  review,  intangible  benefits  of  IT  are 
often  the  most  important  (Senn,  1978:525).  The  costs,  which  are  significantly 
easier  to  quantify,  may  outweigh  the  benefits  of  many  systems  if  the  organization 
is  not  able  to  fully  quantify  the  intangible  benefits. 

Three  of  Strassmann’s  models  may  also  be  useful  for  the  Air  Force.  The 
Residual  Value,  IT  Spending,  and  Cost  of  Information  Management  models  all 
provide  a  means  to  evaluate  how  well  organizations  are  managing  IT  investment. 

The  Residual  Value  and  IT  Spending  models  both  provide  insight  into  how 
well  an  organization  manages  its  IT  investment.  The  Delphi  panel  concurred  that 
the  majority  of  the  variables  required  by  these  two  models  are  either  available  or 
collectable. 
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One  problem  could  develop  with  successful  utilization  of  the  Cost  of 
Information  Management  model.  This  model  is  dependent  on  the  variable  Cost 
of  Sales,  General,  and  Administrative,  and  the  variable  Cost  of  Research  and 
Development.  The  Delphi  panel  was  somewhat  mixed  in  the  availability  or 
collectability  of  the  Cost  of  Sales,  General,  and  Administrative.  The  Delphi  panel 
was  unanimous  that  the  Cost  of  Research  and  Development  was  not-collectable. 
This  model  requires  more  study  before  its  usefulness  can  be  fully  determined. 

The  final  model  that  may  prove  useful  to  the  Air  Force  is  the  Balanced 
Scorecard.  This  model  is  designed  to  use  quantifiable  variables  to  track  how 
well  an  organization  meets  certain  strategic  goals.  While  the  total  value  of  an  IT 
investment  is  not  quantified,  the  model  does  provide  a  complete  framework  for 
managing  IT  investment.  This  may  well  be  where  the  Air  Force  needs  to  focus. 

In  his  book  The  Squandered  Computer.  Strassmann  stated  “alignment  is  the 
delivery  of  required  results”  (Strassmann,  1997:3).  This  could  be  how  the  Air 
Force  could  best  focus  its  IT  investment.  Through  the  use  of  the  Residual  Value 
and  IT  Spending  models,  the  Air  Force  may  be  able  to  track  best  practices 
among  its  organizations  and  develop  specific  benchmarks  for  relative  amounts  of 
IT  spending.  While  it  may  take  time  to  initiate  a  system  to  accomplish  this,  it  may 
be  a  move  in  the  right  direction. 

Overall,  the  Balanced  Scorecard,  Residual  Value  and  IT  Spending  models 
may  contribute  to  more  efficient  management  of  IT  investment.  While  they  do 
not  specifically  quantify  the  value  of  an  IT  investment,  they  do  focus  on 
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alignment  with  strategic  goals.  Proper  alignment  with  strategic  goals  can  serve 
to  reduce  the  risks  of  these  investments.  One  could  argue  that  in  the  public 
sector,  where  organizations  cannot  track  income  against  specific  IT  investment, 
that  strategic  alignment  is  where  government  should  focus. 

Table  13  summarizes  the  models  that  are  useable  for  the  Air  Force.  The 
first  five  models  use  variables  that  are  available  or  collectable.  The  last  two 
models  use  variables  that  may  be  relevant  to  specific  Air  Force  organizations. 


Table  13.  Usable  Models 


Equation  Number 

Name 

3  and  4 

Cost  Benefit  Analysis 

9 

Cost  of  Information  Management 

14 

IT  Spending 

16 

Residual  Value 

18 

Balanced  Scorecard 

8 

Business  Profitability 

13 

Information  Productivity 

Limitations  and  Implications 

The  search  for  models  currently  in  use  was  based  on  models  that  have 
been  published  in  recognized  journals  and  texts.  The  majority  of  these  models 
have  not  been  thoroughly  tested  and  therefore  will  face  some  obstacles  before 
widespread  use  in  the  Air  Force. 

There  are  two  important  implications  for  the  Air  Force  that  developed  from 
this  study.  First,  there  are  models  which  are  useful  immediately  to  improve  the 
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Air  Force’s  IT  investment  process.  However,  of  the  models  that  showed 
promise,  there  is  still  work  to  be  done  before  they  are  accepted  by  a  significant 
number  of  the  decision  makers.  The  most  promising  models  which  are  those 
that  have  some  subjective  measures  embedded  in  them  and  present  results 
related  to  how  we  can  better  manage  the  risks  of  the  IT  investment  process  and 
focus  on  strategic  alignment. 

Areas  for  Future  Research 

The  primary  area  for  future  research  is  to  test  the  models  in  real  world 
situations  and  judge  them  on  the  basis  of  how  they  contribute  to  the  IT 
investment  decision  process.  This  would  involve  extensive  coordination  with  an 
agency  that  would  allow  the  researcher  extended  access  to  its  records  and 
processes. 

Another  area  where  future  research  could  improve  this  situation  would  be 
to  try  to  improve  certain  models  so  that  they  are  more  useful  to  the  Air  Force. 
Many  of  the  models  are  unusable  now  because  of  one  or  two  variables.  There  is 
certainly  the  possibility  that  these  models  could  be  refined  to  more  specifically 
address  the  needs  of  the  public  sector.  As  with  testing  models,  this  research 
would  require  extensive  coordination  between  the  researcher  and  an  agency  to 
see  exactly  what  variables  are  available  and  could  be  exchanged  to  make  a 
model  more  useful  and  relevant  to  the  IT  investment  process  in  the  public  sector. 
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Appendix  A:  Delphi  Panel  Electronic  Mail  Cover  Letter 


Dear  (individual’s  name)  28  July  98 

1 .  I  am  a  graduate  student  at  the  Air  Force  Institute  of  Technology  conducting 
research  into  how  the  Air  Force  evaluates  its  investment  in  information 
technology.  As  you  know,  the  Information  Technology  Management  Reform  Act 
(Clinger-Cohen  Act)  requires  the  government  to  improve  its  investment  decisions 
for  proposed  investment  in  information  technology.  I  am  investigating  what 
models  are  currently  used  in  both  the  public  and  private  sector  to  evaluate 
information  technology  investments  and  the  possible  usefulness  of  these  models 
in  the  Air  Force. 

2.  I  request  your  help  to  choose  someone  on  the  MAJCOM  staff  (who  should 
remain  anonymous  to  me)  who  has  some  experience  in  the  area  of  information 
technology  investment  planning  to  review  the  data  that  the  models  require  and 
concur  or  nonconcur  with  my  judgment  on  the  availability  or  collectability  of  these 
variables.  There  are  two  attached  files  with  this  email.  The  first,  (varlist.doc)  is 
the  instructions  and  an  expanded  variable  list.  The  second  attached  file 
(varlist.xls)  shows  the  list  of  variables  required  by  the  models  in  a  matrix  form. 

3.  After  the  review  of  the  variables  at  the  MAJCOM  level,  please  forward 
varlist.xls  back  to  me  by  7  August  98.  There  will  be  no  attribution  of  any 
feedback  from  any  MAJCOM.  The  numbers  will  be  reported  only  as  a 
percentage  of  the  respondents  that  agreed  with  the  variable  validation. 

4.  I  may  be  reached  at  937-426-3328  or  at  tpeachey@afit.af.mil.  My  thesis 
advisor  is  Major  Bill  Scott  who  may  be  contacted  at  DSN  785-7777  ext.  3323,  or 
at  wscott@afit.af.mil. 

5.  At  your  request,  a  copy  of  this  thesis  will  be  sent  to  you  by  electronic  mail  on 
completion. 

Thanks  for  your  help  in  this  research. 

TODD  A.  PEACHEY,  Captain,  USAF 

Graduate  Student,  Air  Force  Institute  of  Technology 
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Instructions  for  Completion  of  Variable  Validation 


1 .  Each  of  these  variables  is  part  of  a  model  that  is  currently  used  or  proposed 
as  a  technique  to  evaluate  information  technology.  For  these  models  to  be 
useful  for  the  Air  Force,  we  must  have  the  data  or  at  least  be  able  to  collect  it  to 
use  the  model. 

2.  This  survey  is  not  designed  for  you  to  have  to  complete  any  research  on 
whether  or  not  your  unit  collects  this  data.  The  answers  are  to  be  based  solely 
on  your  opinions  and  previous  experiences  as  an  Air  Force  Communications 
Officer.  There  will  be  NO  attribution  of  these  opinions  to  any  unit  or  person.  The 
only  information  that  will  be  reported  is  the  percentage  of  responses  in  each 
category.  The  basis  of  this  research  is  simply  to  find  a  model  that  helps  us  better 
evaluate  our  IT  investments,  not  make  any  statements  about  what  we  may  or 
may  not  have  done  in  the  past. 

3.  The  variable  descriptions  listed  in  the  left  hand  column  are  used  in  different 
models  to  evaluate  information  technology.  Any  symbols  in  the  box  are  notation 
from  the  model.  A  more  detailed  description  of  each  variable  is  available  in  the 
next  section  of  the  instructions. 

3.  The  next  column  of  the  sheet  is  “Available.”  This  box  should  be  checked  if 
the  data  is  currently  collected  by  organizations. 

4.  The  next  column  of  the  sheet  is  “Not-Available.”  This  box  should  be  checked 
if  the  data  does  not  apply  to  any  organizations  in  the  Air  Force  that  you  are 
aware  of. 

5.  The  next  column  of  the  sheet  is  “Collectable.”  This  column  should  be 
checked  if  the  data  could  be  collected  in  a  reasonable  time  frame  and  for  a 
reasonable  cost. 

6.  The  next  column  of  the  sheet  is  “Not-  Collectable.”  This  column  should  be 
checked  if  collection  of  the  data  would  require  too  much  time  and  expense  to 
justify  its  use. 

7.  Place  a  “c”  or  “n”  in  the  final  column  to  represent  concurrence  or 
nonconcurrence.  If  an  “n”  is  placed  in  the  box,  place  another  mark,  other  than 
“x”  in  the  box  you  feel  is  most  appropriate  (i.e.  Available,  Not-Available, 
Collectable,  Not-Collectable) 
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Appendix  B.  Extended  Variable  Definitions 


1 .  expected  return  on  asset  -  the  expected  monetary  return  that  an  asset 
generates,  that  is,  the  cash  flow  that  the  asset  generates  divided  by  the  asset’s 
cost 

2.  expected  investment  in  asset  -  initial  cost  of  asset 

3.  time  to  expiration  of  choice  -  used  when  considering  two  or  more  investments, 
this  is  the  time  between  now  and  when  the  latest  information  technology 
investment  could  be  made 

4.  correlation  coefficient  between  return  on  asset  and  investment  in  asset  - 
numerical  description  of  how  return  on  asset  relates  to  investment  in  asset,  that 
is,  a  positive  one  would  imply  return  on  asset  increases  at  the  exact  same 
percentage  rate  as  the  cost  of  the  asset 

5.  standard  deviation  of  return  on  asset  and  investment  in  asset  -  amount  of 
variability  between  return  on  asset  and  investment  in  asset 

6.  cash  flow  at  end  of  period  -  last  cash  inflow  in  a  specific  time  period  that  an 
asset  generates 

7.  risk  free  rate  of  return  -  rate  that  government  borrows  money  for 

8.  price  of  IT  stock  -  total  investment  in  computer  equipment  plus  three  times  the 
amount  of  labor  necessary  to  operate  it 

9.  value  added  -  any  measure  of  output  (sales,  production,  etc.)  minus  total  firm 
non-labor  expenses 

1 0.  IT  stock  /  value  added  -  price  of  IT  stock  divided  by  value  added 

11.  %  above  or  within  budget  -  actual  expenses  divided  by  budgeted  expenses 

12.  allocation  different  budget  items  -  amount  of  budget  items  that  can  be 
allocated  to  different  expense  accounts  or  user  accounts 

13.  IT  budget  as  a  %  of  turnover  -  new  IT  purchases  divided  by  total  IT  assets 


72 


14.  IT  expenses  per  staff  member  -  total  IT  expenses  divided  by  total  number  of 
staff  members,  that  is  administrative  personnel  not  actively  involved  in  IT 
operations 

15.  financial  benefits  from  selling  to  third  parties  -  incoming  cash  flows 
generated  from  selling  IT  capability  to  customers  outside  the  unit 

16.  financial  evaluation  based  on  Return  On  Investment,  Net  Present  Value, 
Internal  Rate  Return  -  data  such  as  incoming  cash  flows  from  the  IT  asset  and 
the  risk-free  interest  rate 

17.  %  of  business  development  capacity  engaged  in  strategic  projects  -  amount 
of  money  and  personnel  hours  from  the  IT  department  dedicated  to  strategic 
projects  divided  by  total  money  and  personnel  hours  of  the  IT  department 

18.  %  of  applications  managed  by  IT  -  number  of  applications  (MS  Office,  PCS, 
etc.)  software  managed  by  IT  personnel  divided  by  total  number  of  applications 

19.  %  of  applications  delivered  by  IT  -  amount  of  application  software  purchased 
and  installed  by  IT  department  divided  by  total  amount  of  applications 

20.  %  of  in-house  applications  -  amount  of  in-house  applications  developed  by 
IT  department  divided  by  total  amount  of  IT  applications 

21 .  frequency  of  IT  Steering  Committee  Meetings  -  number  of  meetings  per 
month,  quarter,  or  year 

22.  %  of  application  development  and  operations  within  a  Service  Level 
Agreement  (SLA)  -  amount  of  application  development  and  operations  provided 
to  users  through  SLA 

23.  %  of  changes  and  adjustments  made  throughout  different  development 
stages  -  number  of  changes  and  adjustments  (software  development)  in  each 
stage  divided  by  total  number  of  adjustments  and  changes 

24.  number  of  defects  per  function  point  in  first  year  of  production  -  refers  to 
software  development 

25.  number  of  function  points  per  person  per  month  -  refers  to  software 
development 

26.  average  number  of  days  late  in  delivering  software  -  software  development 
for  users 
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27.  average  unexpected  budget  increases  -  refers  to  software  development 

28.  %  of  projects  performed  within  SLA  -  number  of  projects  performed  within 
SLA  divided  by  total  number  of  projects 

29.  %  of  code  reused  -  refers  to  software  development 

30.  %  of  maintenance  activities  -  refers  to  software  development,  maintenance 
of  software  currently  in  use 

31 .  %  unavailability  of  mainframe  -  mainframe  downtime  for  users 

32.  %  unavailability  of  network  -  network  downtime  for  users 

33.  response  times  per  category  of  users  -  response  times  from  mainframe  and 
network 

34.  %  of  jobs  completed  within  set  times  -  jobs  by  users  on  mainframe 
completed  within  set  times  divided  by  total  number  of  jobs  by  users 

35.  %  of  reruns  -  number  of  reruns  of  jobs  by  users  divided  by  total  number  of 
jobs  by  users 

36.  average  time  between  systems  failures  -  mainframe  or  network 

37.  ratio  of  operational  costs  to  installed  MIPS  -  operations  costs  of  mainframe 
computer  divided  by  total  MIPS  (or  network  divided  by  throughput) 

38.  average  lead  time  for  deliveries  -  deliveries  on  purchase  of  software  and 
hardware  purchased  from  outside  vendors 

39.  average  answer  time  of  help  desk  -  network  or  other  computer  related  help 
desks  within  the  unit 

40.  %  of  questions  answered  within  set  time  -  number  of  questions  to  help  desk 
answered  within  set  time  divided  by  total  number  of  questions 

41 .  %  of  solutions  within  SLA  -  number  of  solutions  to  help  desk  problems  within 
SLA  divided  by  total  number  of  help  desk  solutions 
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42.  %  of  users  that  already  received  education  per  application  /  technology  - 
number  of  users  that  have  received  training  in  particular  application  /  technology 
divided  by  total  number  of  users  of  the  application  /  technology 

43.  quality  index  of  education  -  subjective  measurement  tool  designed  to  assess 
quality  of  training 

44.  number  of  hours  that  can  be  charged  internally  or  externally  -  number  of 
hours  of  IT  staff  work  that  can  be  charged  to  specific  users  (maintenance  and 
other  tasks) 

45.  %  of  people  hours  charged  on  projects  -  people  hours  that  can  be  charged 
to  specific  projects 

46.  satisfaction  index  of  IT  staffs  -  measurement  tool  designed  to  assess  IT  staff 
satisfaction  with  their  job 

47.  %  of  IT  staff  that  can  access  groupware  facilities  -  number  of  IT  staff  that 
can  access  groupware  facilities  divided  by  total  number  of  IT  staff 

48.  %  of  IT  staff  that  can  effectively  use  groupware  facilities  -  number  of  IT  staff 
trained  in  use  of  groupware  facilities  divided  by  total  number  of  IT  staff 

49.  number  of  education  days  per  person  -  number  of  training  days  per  IT  staff 
member 

50.  education  budget  as  a  %  of  IT  budget  -  education  budget  in  dollars  divided 
by  total  IT  budget  in  dollars 

51 .  number  of  years  experience  per  IT  staff  member  -  specific  number  of  years 
experience  of  each  IT  staff  member  used  to  compute  average  IT  staff  experience 

52.  age  pyramid  of  IT  staff  -  age  data  on  IT  staff  used  to  measure  IT  staff 
experience 

53.  number  of  applications  per  age  category  -  age  of  application  software 
ranked  into  specific  categories 

54.  number  of  applications  younger  than  five  years  -  age  of  software 

55.  %  of  budget  spent  on  IT  research  -  IT  budget  dollars  spent  on  research 
divided  by  total  IT  budget 
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56.  hardware  costs  -  total  costs  of  hardware  including  purchase,  installation, 
maintenance,  etc. 

57.  software  costs  -  total  costs  of  software  including  purchase,  installation, 
maintenance,  etc. 

58.  transmission  costs  -  total  costs  of  connectivity  including  purchase, 
installation,  maintenance,  etc. 

59.  processing  costs  savings-  total  costs  associated  with  faster  processing 
associated  with  a  new  system 

60.  paper  savings  -  reduced  costs  from  new  systems 

61.  inventory  savings  -  savings  from  reduced  inventory  due  specifically  to  new 
system 

62.  quantifiable  intangible  benefits  -  any  benefit  from  a  new  system  that  is 
quantifiable 

63.  software  maintenance  costs  -  any  cost  related  to  software  maintenance 

64.  fixed  annual  costs  of  new  system  -  any  cost  that  can  be  traced  to  the  new 
system  on  an  annualized  basis 

65.  cost  of  sales,  general  and  administrative  -  all  costs  traceable  to  sales  and 
administrative  functions 

66.  cost  of  research  and  development  -  all  costs  traceable  to  an  organization’s 
research  and  development  efforts 

67.  shareholder  equity  -  standard  ownership  of  corporations 

68.  capital  surplus  -  excess  capital  after  dividends  paid  to  shareholders 

69.  management  value  added  -  funds  left  over  after  all  costs  are  accounted  for 

70.  costs  of  management  -  expenses  not  traceable  to  operations  activities 

71 .  costs  of  information  management  -  costs  traceable  to  information 
management  activities 
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72.  value  added  by  information  -  net  profit  minus  financial  capital  assets  times 
risk  free  interest  rate 

73.  information  productivity  -  value  added  by  information  divided  by  cost  of 
information  management  (variable  72  /  variable  71) 

74.  equipment  depreciation  -  loss  in  value  of  equipment  (computers,  routers, 
etc.)  through  use  and  advancement  of  technology 

75.  development  depreciation  -  loss  in  value  of  equipment,  software,  etc. 
through  use  or  advancement  of  technology  (refers  to  in-house  development) 

76.  software  depreciation  -  loss  in  value  of  software  due  to  technology 
advancement 

77.  training  depreciation  -  loss  in  value  of  training  due  to  new  equipment, 
software,  turnover  of  personnel  etc. 
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